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ABSTRACT 


Budgetary  concerns  over  the  last  decade  have  put  increased  pressure  on  federal 
agencies  to  improve  efficiency  and  create  cost  savings.  Accordingly,  the  Government 
Accountability  Office  (GAO)  and  other  watchdog  groups  have  increasingly  scrutinized 
government  source  selections;  GAO  reports  and  procurement  experts  alike  indicate 
opportunities  for  improvement  in  this  area.  To  aid  in  this  improvement  initiative,  our 
research  focuses  on  the  contract  management  process,  with  special  emphasis  on  the 
source  selection  methods  of  tradeoff  and  lowest  price  technically  acceptable  (LPTA). 
Specifically,  our  data  analysis  explores  the  relationship  of  source  selection  methods  to  the 
contract  outcomes  of  procurement  administrative  lead  times  (PALT)  and  contractor 
performance  assessment  reporting  system  (CPARS)  ratings.  The  results  of  our  analysis 
showed  no  statistically  significant  relationship  between  source  selection  method  and 
contract  outcomes.  However,  other  variables,  namely  the  number  of  evaluation  factors 
and  number  of  offers  received,  were  shown  to  have  a  significant  effect  on  PALT.  At  the 
conclusion  of  this  MBA  Professional  Report,  we  present  suggestions  for  further  research 
to  build  upon  these  findings. 
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I.  INTRODUCTION 


A.  BACKGROUND 


The  Department  of  Defense  (DOD)  accounts  for  a  substantial  percentage  of  the 
federal  budget  each  year  (Albano,  2013).  Therefore,  DOD  acquisition  professionals  must 
focus  on  achieving  best  value  for  the  government  through  pursuit  of  cost  savings,  while 
also  adhering  to  stated  regulations.  In  the  past,  Government  Accountability  Office  (GAO) 
reports,  professional  journals,  and  acquisition  magazines  have  identified  a  wide  disparity 
within  source  selection  processes;  while  some  are  best  in  class,  others  consistently  fail  to 
achieve  the  desired  outcomes.  For  example,  strengths  of  current  source  selection  processes 
include  the  use  of:  engineering  procedures  in  strategy  definition,  Integrated  Project  Teams 
(IPTs),  Performance-Based  Acquisition  (PBA),  and  good  communication  techniques.  On 
the  other  hand,  risk  assessments,  source  selection  evaluation  factors,  schedule  slips,  and 
cost  overruns  are  all  areas  of  significant  weakness  (Maser  &  Thompson,  2013).  In  1992, 
the  GAO  highlighted  DOD  contract  management  as  high-risk;  over  two  decades  later,  it  is 
still  a  point  of  major  concern  (Albano,  2013).  Moreover,  the  DOD  inspector  general 
identified  142  different  deficiencies  relating  to  contract  management  (Albano,  2013). 

A  more  recent  GAO  (2010)  report  highlighted  training,  education,  and  experience 
level  of  the  acquisition  workforce  as  areas  of  concern.  Each  of  these  topics  represents  an 
opportunity  for  improvement  and  savings,  particularly  initiatives  focused  on  education  and 
training.  However,  this  report  only  highlights  a  fraction  of  the  potential  cost  savings  within 
federal  acquisition.  As  such,  GAO,  DOD,  and  Air  Force  Installation  Contracting  Agency 
(AFICA)  investigators  have  all  engaged  in  a  thorough  analysis  of  previous  procurements  in 
an  attempt  to  identify  areas  for  future  savings.  AFICA  is  an  Air  Force  organization 
designed  to  analyze  purchasing  data  and  ultimately  develop  strategies  for  strategic  sourcing 
(R.  Westermeyer,  personal  communication,  May  15,  2015).  Strategic  sourcing  is  an 
institutional  level  of  acquisition  where  products  and  services  are  purchased  at  the  enterprise 
level  to  drive  down  costs  through  economies  of  scale,  demand  management,  and  supplier 
integration  (R.  Westermeyer,  personal  communication,  May  15,  2015). 
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The  steps  of  the  contract  management  process,  which  can  be  optimized  to  produce 
cost  savings,  are:  procurement  planning,  solicitation  planning,  solicitation,  source  selection, 
contract  administration,  and  contract  closeout  (Garrett,  2007);  however,  our  research 
focuses  primarily  on  the  activities  prior  to  contract  award.  Furthermore,  the  source 
selection  process  can  be  broken  down  into  two  contracting  methods:  sealed  bidding  and 
negotiations.  Typically,  sealed  bidding  is  reserved  for  simple,  well-defined  requirements. 
The  alternative,  contracting  by  negotiation,  is  utilized  for  those  acquisitions  in  which 
requirements  are  not  simple  or  well-defined. 

Contracting  by  negotiation  is  discussed  in  FAR  15.101  and  is  further  broken  down 
into  two  source  selection  methods.  The  first,  tradeoff,  allows  the  government  the  flexibility 
to  weigh  non-cost  factors  as  part  of  an  overall  evaluation  assessment.  The  second,  lowest 
price  technically  acceptable  (LPTA),  requires  the  government  to  first  rank  all  offers  in 
reverse  price  order,  then  review  them  in  that  order  until  a  technically  acceptable  offer  is 
identified  (FAR  15.101-2).  Each  acquisition  is  unique,  and  the  appropriate  method  can  only 
be  selected  after  reviewing  the  complexity  of  the  program  and  all  pertinent  risks  (Shultz, 
2015). 

B.  PROBLEM  STATEMENT 

Currently,  there  is  little  empirical  evidence  identifying  a  relationship  between 
source  selection  method  and  contract  outcomes.  This  lack  of  reliable  data  on  the  subject  has 
forced  contracting  professionals  to  rely  heavily  on  experience  and  anecdotal  evidence, 
which  can  lead  to  confusion.  Consider,  for  example,  the  terminology  involved  in  the 
discussion.  Despite  the  specificity  of  the  definitions  provided  in  the  Federal  Acquisition 
Regulation  (FAR),  many  top  analysts — some  of  whom  have  published  works  on  the 
topic — still  confuse  key  terms.  For  example,  tradeoff  is  often  confused  with  best  value,  the 
overarching  term  describing  the  entire  continuum  of  options  available  to  contracting 
officers  charged  with  ensuring  greatest  benefit  to  the  government.  This  error  is  most 
prevalent  in  the  private  sector;  however,  even  the  Flonorable  Jacques  Gansler — fonner 
Undersecretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USDAT&L) — 
muddles  the  terms  in  a  paper  published  for  a  U.S.  Senate  subcommittee  (Levin,  2014). 
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Identification  of  this  problem  is  not  meant  to  be  accusatory;  instead,  our  aim  is  to  recognize 
the  need  for  empirical  data  that  eliminates  confusion  on  the  subject  of  source  selection 
methods  and  their  related  outcomes. 

C.  PURPOSE  STATEMENT 

The  purpose  of  our  research  is  to  empirically  identify  key  relationships  between  the 
source  selection  methods  and  measurable  contract  outcomes.  The  motivation  for  our 
research  stems  from  the  lack  of  empirical  evidence  regarding  this  relationship,  as  well  as 
from  a  desire  to  improve  contract  outcomes.  Unnecessarily  long  lead  times,  protests,  vague 
specifications,  and  cost  and  schedule  overruns  all  plague  defense  acquisition.  Ideally,  an 
acquisition  should  strive  to  keep  procurement  acquisition  lead  time  (PALT)  as  low  as 
practical  and  Contractor  Performance  Assessment  Reporting  System  (CPARS)  ratings  as 
high  as  possible.  Positive  contract  outcomes  are  rooted  in  the  procurement  planning  phase; 
it  is  in  this  phase  that  the  procurement  method,  source  selection  method,  and  contract  type 
are  decided.  Our  research  discusses  the  relationship  of  outcome  variables  to  the  chosen 
source  selection  method,  with  the  ultimate  goal  of  better  understanding  the  implications  of 
this  decision  facing  acquisition  teams. 

D.  RESEARCH  QUESTIONS 

In  this  research,  we  answer  the  following  questions: 

1 .  What  is  the  relationship  between  the  source  selection  method  (LPTA  versus 
tradeoff)  and  CPARS  ratings? 

2.  What  is  the  relationship  between  the  source  selection  method  (LPTA  versus 
tradeoff)  and  PALT? 

3.  What  percentage  of  procurements  using  the  tradeoff  source  selection 
method  is  awarded  based  on  non-price  factors? 

E.  RESEARCH  BENEFITS  AND  LIMITATIONS 

The  principle  benefit  of  our  research  is  that  it  provides  data  relating  to  source 
selection  methods  and  their  subsequent  contract  outcomes.  We  accessed  contract  files 
containing  all  data  necessary  for  complete  statistical  analysis.  Historically,  there  has  been  a 
lack  of  empirical  data  on  this  subject;  our  research  and  analysis  addresses  this  gap. 
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The  main  limitation  of  our  research  is  the  sample  size  of  our  data.  Time  and  travel 
constraints  limited  our  data  collection  to  include  the  contract  files  from  only  two  Air  Force 
bases,  as  we  discuss  further  in  Chapter  III.  To  broaden  the  scope  of  the  research  and  reach 
more  definitive  conclusions,  we  would  ideally  have  been  able  to  access  more  contract  files 
from  additional  installations,  including  those  from  defense  branches  other  than  the  Air 
Force. 

F.  ORGANIZATION  OF  REPORT 

This  first  chapter  presented  the  background,  problem  statement,  purpose  statement, 
research  questions,  and  benefits  and  limitations  of  our  research.  In  Chapter  II,  we  provide  a 
review  of  relevant  literature  regarding  source  selection  methods,  including  guiding 
principles  for  the  federal  acquisition  system,  significant  defense  acquisition  reform 
initiatives,  an  overview  of  the  contract  management  process,  pre-award  acquisition 
decisions,  and  a  description  of  each  source  selection  method  (LPTA  and  tradeoff)  and  their 
corresponding  implications.  In  Chapter  III,  we  explain  the  methodology  used  to  conduct 
our  research,  which  includes  a  discussion  of  Air  Force  contracting  structure  and  systems 
together  with  a  description  of  our  data  collection  process  and  its  unique  elements.  In 
Chapter  IV,  we  describe  our  data,  present  the  results,  discuss  our  analysis,  and  make 
recommendations  based  on  the  findings.  In  Chapter  V,  we  summarize  our  findings,  present 
our  conclusion,  answer  the  research  questions,  and  suggest  areas  for  further  research. 

G.  SUMMARY 

This  chapter  served  as  an  introduction  to  our  research,  providing  a  brief  background 
of  source  selections  and  their  role  in  the  acquisition  environment.  It  also  stated  the  problem 
to  be  addressed,  the  purpose  of  our  research,  and  the  research  questions  to  be  answered. 
Lastly,  we  stated  the  benefits  and  limitations  of  the  research.  In  the  next  chapter,  we 
examine  academic  journals,  GAO  reports,  and  other  source  selection  texts  in  order  to 
review  the  literature  and  increase  understanding  of  federal  contract  management. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  present  a  review  of  the  literature  concerning  the 
federal  contract  management  process,  specifically  regarding  the  role  of  source  selection 
methods  available  to  contracting  officers.  To  ensure  a  comprehensive  understanding  of 
all  relevant  sources,  the  review  incorporates  information  from  federal  regulations,  the 
Government  Accountability  Office  (GAO),  and  prominent  contracting  journals. 

First,  we  cover  the  guiding  principles  for  the  federal  acquisition  system,  followed 
by  significant  defense  acquisition  reform  initiatives.  Next,  we  discuss  the  government 
contract  management  process,  before  moving  to  pre-award  acquisition  decisions.  Finally, 
we  will  explain  the  two  source  selection  methods,  lowest  price  technically  acceptable 
(LPT A)  and  tradeoff,  as  well  as  implications  of  each. 

B.  GUIDING  PRINCIPLES  FOR  THE  FEDERAL  ACQUISITION  SYSTEM 

Federal  contracting  is  very  different  from  commercial  purchasing.  A  wide  array  of 
statutory  and  agency-instituted  regulations  restricts  the  actions  of  federal  contracting 
officers,  chief  of  which  is  the  Federal  Acquisition  Regulation  (FAR).  This  body  of 
regulation  provides  instruction  on  the  entire  contract  management  process,  from  the 
initiation  of  procurement  planning  to  the  final  steps  of  contract  closeout.  The  FAR  is  a 
living  document,  which  regulators  still  revise  to  this  day,  as  new  policies  are  introduced. 
However,  regulators  realize  that  the  diverse  scenarios  contracting  officers  face  far  exceed 
the  capacity  of  a  single  regulation.  Therefore,  the  FAR  instructs  acquisition  personnel  to 
consider  actions  approved  if  not  explicitly  prohibited: 

In  exercising  initiative,  Government  members  of  the  Acquisition  Team 
may  assume  if  a  specific  strategy,  practice,  policy  or  procedure  is  in  the 
best  interests  of  the  Government  and  is  not  addressed  in  the  FAR,  nor 
prohibited  by  law  (statute  or  case  law),  Executive  order  or  other 
regulation,  that  the  strategy,  practice,  policy  or  procedure  is  a  permissible 
exercise  of  authority.  (FAR  1.102(d)) 
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The  vast  size  of  the  federal  government  necessitates  additional  regulations  to  help 
establish  policy.  Subordinate  to  the  FAR  are  various  agency  supplements,  with  the  DOD 
maintaining  the  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS),  which  is 
one  of  the  more  robust  versions.  Each  of  the  service  branches  has  added  another  level  of 
policy  below  the  DFARS,  and  most  contracting  activities  have  issued  several  layers  of 
policy  memoranda  below  each  service-specific  supplement.  However,  as  our  research 
focuses  on  defense  procurement,  our  analysis  does  not  extend  to  the  level  of  detail 
beyond  the  DFARS,  therefore  we  have  not  included  service  supplements. 

In  addition  to  describing  which  actions  are  not  permitted,  the  FAR  explains  the 
four  basic  principles  of  govermnent  procurement.  We  have  listed  each  below. 

1.  Satisfy  the  Customer  in  Terms  of  Cost,  Quality,  and  Timeliness  of  the 
Delivered  Product  or  Service 

Ultimately,  the  role  of  federal  procurement  is  to  augment  organic  government 
personnel  and  equipment  in  order  to  carry  out  the  mission  or  function  of  the  end  user.  In 
the  DOD,  the  end  user  is  typically  the  warfighter,  and  all  acquisition  actions  should  be 
structured  to  provide  the  necessary  level  of  support  for  our  nation’s  warriors,  whether  in 
garrison  or  abroad.  Additionally,  the  FAR  expands  this  concept  by  directing  contracting 
officers  to  (1)  use  commercial  items  whenever  possible,  (2)  conduct  business  with 
vendors  that  can  demonstrate  competence,  and  (3)  seek  out  competition. 

2.  Minimize  Administrative  Operating  Costs 

Awarding  a  contract  to  a  vendor  is  only  one  step  in  the  contract  management 
process.  Depending  on  the  size  of  the  purchase,  several  weeks  or  even  months  of  market 
research  and  source  selection  procedures  may  have  taken  place  beforehand.  Likewise, 
modifications,  tenninations,  and  contractor  surveillance  tasks  can  extend  years  beyond 
the  effective  date  of  a  contract.  Government  officials  should  take  every  measure  to 
streamline  the  processes  before  and  after  award  in  order  to  remain  good  stewards  of 
taxpayer  dollars. 
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3.  Conduct  Business  with  Integrity,  Fairness,  and  Openness 

Unlike  profit-seeking  firms,  which  are  responsible  only  to  shareholders  or  owners, 
federal  institutions  have  additional  mandates  because  of  their  reliance  on  taxpayer 
dollars.  One  such  mandate  is  the  requirement  for  transparency,  which  ranges  from 
publicizing  new  solicitations  online  to  fielding  Freedom  of  Information  Act  requests  on 
existing  contracts. 

4.  Fulfill  Public  Policy  Objectives 

The  final  objective  of  government  procurement  is  a  set  of  policy  agenda  items 
established  by  Congress,  which  are  not  meant  to  directly  affect  the  warfighter  or  the 
contracts  themselves.  The  most  prominent  example  is  the  small  business  set-aside 
program,  which  ensures  that  small,  disadvantaged  firms  receive  a  portion  of  government 
contracts.  In  many  cases,  accommodating  these  goals  often  comes  at  the  expense  of  price 
or  schedule,  although  proponents  argue  that  small  companies  are  just  as  capable  as  are 
established  defense  firms.  This  objective  of  meeting  Congressional  agenda  items  through 
procurement  was  not  an  original  goal  of  the  federal  government,  but  policymakers 
introduced  and  subsequently  modified  the  objective  later  over  the  course  of  a  series  of 
acquisition  reforms.  The  following  is  a  brief  summary  of  those  reforms. 

C.  SIGNIFICANT  DEFENSE  ACQUISITION  REFORM  INITIATIVES 

As  the  principles  described  in  the  previous  section  indicate,  simultaneously 
achieving  all  federal  procurement  goals  is  typically  not  feasible.  Instead,  contracting 
officers  frequently  weigh  different  priorities  against  each  other,  resulting  in  a  balance  of 
results  that  vary  among  acquisitions.  Originally,  the  government  was  chiefly  concerned 
with  receiving  an  efficient  price  in  a  generally  equitable  way  (Manuel,  2011);  over  time, 
however,  the  objectives  of  public  policy  have  changed.  The  following  is  a  list  of  relevant 
legislation,  along  with  an  indication  of  which  goal(s)  they  support. 

1.  Armed  Services  Procurement  Act  of  1947 

The  Armed  Services  Procurement  Act  (ASPA)  of  1947  represents  the  DOD’s  first 
real  departure  from  competitive  acquisition  during  peacetime  operations.  Starting  in  the 
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18th  century,  federal  regulations  called  for  open  solicitation  of  federal  requirements, 
relying  on  market  forces  to  obtain  the  best  price  and  safeguard  taxpayer  dollars. 
However,  after  experiencing  the  increased  demands  of  both  world  wars,  officials  realized 
that  some  items  contained  rigid  delivery  or  performance  requirements  that  made 
competition  problematic  or  outright  infeasible  (Manuel,  2011).  The  ASPA  applied  only 
to  defense  agencies  and  still  called  for  competition,  but  it  included  17  exemption 
categories,  allowing  contracting  officers  much  more  latitude  in  procuring  critical  items. 

2.  Small  Business  Act  of  1953 

While  the  Small  Business  Act  (SBA)  was  passed  in  1953,  the  Hoover 
administration  had  laid  the  foundation  more  than  two  decades  earlier  (Bail,  2010).  After 
the  stock  market  crash  of  1929,  President  Hoover  fonned  the  Reconstruction  Finance 
Corporation  to  loan  government  funds  to  businesses,  with  the  hope  of  jump-starting  the 
economy.  Over  the  next  24  years,  the  organization  continued  to  evolve,  giving  rise  to  the 
Small  War  Plant  Corporation  and  Small  Defense  Plant  Corporation  during  World  War  II 
and  the  Korean  War,  respectively.  The  focus  of  the  effort  shifted  from  stimulating  the 
economy  to  retaining  wartime  production  capacity  to  ultimately  guaranteeing  that  small 
businesses  remained  competitive  in  future  government  contracts. 

Interestingly,  the  SBA  has  changed  relatively  little  in  the  62  years  since  its 
passage,  considering  the  number  of  changes  made  to  competition  and  commercial 
acquisition  policy.  For  example,  the  original  act  provided  for  small  business  size 
standards  that  were  specific  to  each  agency,  formed  the  Small  Business  Administration  to 
help  small  business  concerns,  and  focused  on  awarding  contracts  to  small  businesses, 
“even  if  awards  were  made  at  a  higher  rather  than  lower  price”  (Bail,  2010,  p.  80).  One 
significant  revision  came  in  1978  when  the  federal  government  opted  to  reserve  awards 
below  $25,000  to  small  businesses — the  precursor  to  today’s  small  business  set-asides. 

3.  Truth  in  Negotiations  Act  of  1962 

After  passing  the  ASPA,  Congress  closely  monitored  the  effectiveness  of 
negotiated  contracts.  During  the  1950s,  several  GAO  reports  confirmed  the  military  had 
overpaid  by  approximately  $61  million  in  a  five-year  span  (Williams,  1970).  While  the 
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DOD  initially  proposed  an  internal  regulation  scheme,  Congress  overruled  the  DOD’s 
plan  and  passed  the  Truth  in  Negotiations  Act  (TINA),  requiring  contractors  to  submit 
and  certify  cost  or  pricing  information  when  proposals  exceed  a  specific  threshold.  The 
dollar  limit  has  changed  several  times  over  the  last  several  decades,  as  well  as  specific 
provisions  of  the  law  itself  (Calhoon  &  Sybert,  2012).  However,  the  intent  of  the  law 
remains  the  same  as  when  originally  passed — ensuring  government  negotiators  have  the 
required  information  to  effectively  negotiate  with  contractors. 

4.  Competition  in  Contracting  Act  of  1984 

While  the  TINA  helped  improve  negotiated  contracts,  legislators  still  felt  that  a 
return  to  open  competition  was  in  the  best  interest  of  the  government.  However,  instead 
of  returning  to  total  competition,  drafters  agreed  on  full  and  open  as  opposed  to  maximum 
competition  (Manuel,  2011).  The  fonner  term  describes  competition  as  contracting 
officers  understand  it  today:  reasonable  attempts  to  include  vendors  capable  of 
perfonning  as  required  in  the  contract.  Maximum  competition,  an  all-out  attempt  to 
receive  as  many  bids  as  possible,  is  simply  impractical,  and  ultimately  does  not  result  in 
additional  cost  savings.  While  the  Competition  in  Contracting  Act  (CICA)  also  includes 
provisions  for  exemption,  its  total  focus  on  competing  contracts  continues  to  influence 
the  decisions  of  contracting  officers  today. 

Interestingly,  the  CICA  also  formalized  the  concept  of  best  value,  introducing  the 
tradeoff  method  in  the  process.  A  focus  on  keeping  costs  low  led  to  the  predominant  use 
of  sealed  bidding  to  acquire  government  products  and  services  (Lohfeld,  2015).  The 
CICA’s  importance  allowed  contracting  officers  to  consider  different  evaluation  factors 
such  as  management  or  past  performance  in  addition  to  price,  even  allowing  the 
government  to  weigh  them  as  significantly  more  important  than  price. 

5.  Defense  Acquisition  Workforce  Improvement  Act  of  1990 

Despite  the  implementation  of  the  CICA,  examples  of  unreasonably  high 
acquisitions  persisted,  with  key  inquiries  led  by  the  Packard  Commission  (Gates  et  ah, 
2008).  Additionally,  the  growing  amount  of  legislation  from  Congress  and  regulation 
from  each  of  the  component  services  required  a  dedicated  acquisition  workforce  with 
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extensive  training  on  mandatory  policies.  In  1990,  Congress  passed  the  Defense 
Acquisition  Workforce  Improvement  Act  (DAWIA),  which  (1)  formed  the  Defense 
Acquisition  University  (DAU),  (2)  directed  the  services  to  create  formal  acquisition 
career  paths,  and  (3)  led  to  the  implementation  of  several  DOD  instructions  that  continue 
to  tailor  the  acquisition  workforce  (Gates  et  ah,  2008).  While  previous  legislation  focused 
on  contracting  policy,  the  DAWIA  was  drafted  specifically  to  shape  and  monitor  the 
skills  and  capabilities  of  acquisition  personnel. 

6.  Federal  Acquisition  Streamlining  Act  of  1994 

A  mere  four  years  after  the  implementation  of  the  DAWIA,  the  federal 
government  again  decided  to  alter  the  nature  of  defense  acquisition  in  a  sweeping  way. 
During  the  early  1990s,  the  Clinton  administration  led  a  major  effort  to  review  and 
respond  to  the  cost  of  federal  procurement,  which  was  growing  at  an  alarming  rate 
(Barry,  1995).  The  research  concluded  that  government-specific  regulations  made  up  a 
significant  portion  of  contract  costs;  the  TINA  alone  “add[ed]  about  a  20  percent 
premium  to  govermnent  contracts”  (Barry,  1995,  p.  124).  For  some  firms,  this  complex 
web  of  regulations  was  simply  not  worth  the  effort,  and  they  declined  to  do  business  with 
the  government  altogether  (Barry,  1995). 

In  response,  the  Federal  Acquisition  Streamlining  Act  (FASA)  aimed  to  harness 
the  efficiency  of  the  private  sector  by  shifting  to  commercial  specifications  and  relying 
on  industry  standards  and  market  forces  to  ensure  the  best  prices.  To  this  end,  the  law 
created  a  preference  for  commercial  products,  established  electronic  posting  of 
requirements,  and  allowed  additional  exemptions  for  contingency  contracting — especially 
relevant  after  lessons  learned  during  the  first  Gulf  War  (Barry,  1995).  Additionally,  the 
FASA  mandated  awards  to  small  business  concerns  for  purchases  below  the  simplified 
acquisition  threshold,  but  above  the  newly  created  micro-purchase  threshold.  Finally,  the 
FASA  created  what  we  now  know  as  simplified  acquisition  procedures,  through  a 
complete  rewrite  of  FAR  Part  13  and  the  repeal  of  225  procurement  statutes  (Barry, 
1995). 
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7.  Clinger-Cohen  Act  of  1996 

Originally  passed  as  Divisions  D  and  E  of  the  1996  National  Defense 
Authorization  Act,  the  Clinger-Cohen  Act  (CCA)  included  fixes  to  several  FASA 
provisions,  as  well  as  enacted  new  policies  (Sherman,  1997).  Division  D,  originally 
known  as  the  Information  Technology  Management  Refonn  Act  (ITMRA), 
fundamentally  changed  the  way  the  DOD  acquired  information  technology  (IT)  services 
by  removing  the  General  Services  Administration’s  (GSA’s)  role  as  manager  of  IT 
acquisitions.  Additionally,  the  ITMRA  directed  each  service  to  create  chief  information 
officer  (CIO)  position  to  consolidate  and  report  on  IT  issues. 

Division  E,  also  known  as  the  Federal  Acquisition  Refonn  Act  (FARA),  included 
several  significant  changes  as  well.  The  FARA  refined  the  concept  of  competitive  range 
by  allowing  contracting  officers  “to  limit  the  size  of  the  competitive  range  to  the  greatest 
number  of  proposals  that  will  pennit  efficient  competition”  (Rumbaugh,  2010,  p.  24). 
This  update  considers  the  timeline  of  the  contract  management  process  in  addition  to 
transparency  concerns;  acquisition  teams  now  have  the  flexibility  to  determine  the 
appropriate  priorities  for  each  procurement.  The  FARA  also  introduced  FAR  13.5,  which 
was  referred  to  as  the  Test  Program  for  Commercial  Items,  and  has  recently  become  a 
permanent  addition  to  the  FAR  (Sherman,  1997).  One  interesting  note  about  the  FARA  is 
the  timeline  of  its  enactment — agency  officials  had  not  yet  finished  implementing  the 
Federal  Acquisition  Streamlining  Act  (FASA).  One  analyst  dubbed  the  entire  process 
“Reforming  [the]  Refonn”  (Sherman,  1997,  p.  1),  as  the  flurry  of  new  legislation 
outpaced  the  agency’s  ability  to  execute  them. 

8.  Weapon  Systems  Acquisition  Reform  Act  of  2009 

Just  over  a  decade  after  the  Clinger-Cohen  Act,  Congress  passed  the  Weapon 
Systems  Acquisition  Refonn  Act  (WSARA),  which  deals  specifically  with  Major 
Defense  Acquisition  Programs  (MDAPs).  With  the  overall  goal  of  reducing  cost  and 
schedule  overruns — estimated  at  $3.1  billion  and  22  months  per  program — it  also  made 
improvements  to  contract  performance  monitoring  (Berteau,  Hofbauer  &  Sanok,  2010). 
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To  this  end,  the  WSARA  created  four  new  oversight  positions,  added  more  stringent 
criteria  for  advancing  milestones,  and  supplemented  reporting  requirements  to  Congress. 

As  the  record  of  legislation  shows,  the  DOD  has  been  tweaking  acquisition 
procedures  for  decades,  with  no  end  in  sight.  While  each  change  represents  an 
incremental  improvement  in  the  acquisition  process,  regulators  and  industry  players  agree 
that  more  work  remains.  Of  note,  each  piece  of  legislation  ultimately  rebalanced  the  four 
principles  of  the  federal  acquisition  system. 

9.  Better  Buying  Power 

The  most  recent  piece  of  acquisition  reform,  however,  did  not  originate  from 
Congress.  The  creation  of  Secretary  of  Defense  Ashton  Carter  and  USD(AT&L)  Frank 
Kendall,  Better  Buying  Power  (BBP)  was  originally  a  re-emphasis  of  existing  policies, 
which  ultimately  grew  into  a  unique  piece  of  refonn  (Kendall,  2014).  When  asked,  Carter 
explained  that  the  concept  behind  the  improvements  was  always  an  iterative  process;  no 
one  initiative  can  succeed  in  a  vacuum  (Kendall,  2014).  Now  in  its  third  installment,  the 
program  lives  on,  building  upon  the  same  core  principles  that  started  five  years  ago 
(Kendall,  2010).  One  of  those  principles  is  the  concept  of  affordability.  The  first  version 
of  the  initiative  stressed  the  importance  of  “targeting  affordability  and  controlling  cost 
growth”  (Kendall,  2010),  while  the  update  two  years  later  highlighted  the  need  to 
“mandate  affordability  as  a  requirement”  (Kendall,  2012).  In  the  latest  update,  the  focus 
has  narrowed  to  “enforce  affordability  caps”  (Kendall,  2014),  but  the  emphasis  on 
controlling  costs  is  still  present. 

However,  the  importance  of  affordability  cannot  come  at  the  expense  of  other 
acquisition  objectives.  As  reformers  are  quick  to  point  out,  the  goal  is  process 
improvement,  not  to  “cut  a  percentage  off  the  contractor’s  bid”  (Hagen,  2014).  Likewise, 
cheaper  programs  cannot  come  at  the  expense  of  warfighter  capability.  In  fact,  Carter  and 
Kendall  designed  the  original  set  of  BBP  initiatives  around  the  opposite  premise:  the 
ultimate  goal  is  to  direct  savings  into  operational  accounts  (Corin,  2013).  Instead  of 
buying  fewer  or  cheaper  items,  the  architects  of  BBP  understood  the  need  to  buy  smarter. 
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But  what  does  buying  smarter  look  like?  In  order  to  answer  this  question,  it  is 
important  to  understand  how  the  government  currently  acquires  goods  and  services.  The 
following  is  a  brief  discussion  of  the  contract  management  process. 

D.  GOVERNMENT  CONTRACT  MANAGEMENT  PROCESS 

Regardless  of  the  good  or  service,  government  contracting  typically  follows  the 
same  timeline  of  six  basic  steps  (Garrett,  2007): 


procurement  planning 
solicitation  planning 
solicitation 
source  selection 
contract  administration 

contract  closeout 


Each  of  these  steps  represents  a  broad  overview  of  fundamental  functions,  and 
can  be  as  detailed  or  a  brief  as  necessary.  For  example,  an  order  for  common  printer  toner 
certainly  requires  less  contract  administration  than  a  multi-million-dollar  construction 
effort.  Nevertheless,  each  of  the  above  steps  is  present  in  virtually  all  government 
acquisitions,  even  if  only  briefly. 

1.  Procurement  Planning 

The  first  step  in  the  contract  management  process  is  procurement  planning. 
During  this  step,  the  buyer  contemplates  whether  contracting  is  even  necessary,  often 
categorized  as  a  make-or-buy  decision  (Garrett,  2007).  While  the  government  seldom 
produces  physical  goods  organically,  unifonned  personnel  are  capable  of  performing 
many  federal  services.  In  these  cases,  analysts  conduct  internal  cost-benefit  analyses  to 
determine  the  best  approach  to  fulfill  the  need.  Additionally,  this  step  typically  includes 
market  research  to  define  the  need  itself,  as  well  as  to  understand  the  capabilities  of 
potential  bidders  (Garrett,  2007).  The  implication  of  this  step  should  not  be  understated. 
Procurement  planning  sets  the  foundation  for  three  key  decisions:  contract  type  (fixed- 


ACQUIS1TION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  13- 


price  versus  cost-reimbursement),  procurement  method  (sealed  bidding  versus  contract 
by  negotiation),  and  source  selection  method  (LPTA  versus  tradeoff). 

2.  Solicitation  Planning 

Once  the  acquisition  team  decides  to  proceed  with  a  contract,  solicitation  planning 
begins.  During  this  step,  the  government  documents  the  details  of  the  contract,  from 
evaluation  criteria  to  length  of  the  relationship  with  the  contract  winner  (Garrett,  2007). 
While  government  officials  must  operate  within  the  confines  of  procurement  regulations 
throughout  the  process,  policymakers  have  developed  various  acquisition  techniques  as 
part  of  the  ongoing  reform  previously  mentioned.  Officials  have  introduced  each  new 
strategy  in  order  to  assist  acquisition  teams  in  procuring  the  right  item  in  a  fair  and 
transparent  manner.  We  discuss  these  in  detail  later;  however,  it  is  important  to  note  that 
like  procurement  planning,  this  step  has  a  significant  impact  on  the  success  of  the 
contract. 


3.  Solicitation 

Solicitation  formally  begins  once  the  government  publicizes  the  request  for  goods 
or  services.  As  firms  review  solicitation  documents  and  present  offers  to  the  government, 
officials  must  be  prepared  to  answer  questions,  provide  additional  information  on  the 
requirement,  and  amend  documents  as  necessary.  One  important  note  concerning 
solicitation  is  the  efficiencies  gained  by  recent  technological  developments.  For  example, 
the  introduction  of  FedBizOpps,  the  online  government  point  of  entry,  allows  contracting 
offices  to  publicize  notices  instantly  from  a  computer,  rather  than  mailing  out  dozens  of 
packages  to  prospective  bidders. 

While  a  poorly  conducted  solicitation  can  negatively  affect  the  contract 
management  process,  successful  completion  of  this  step  hinges  on  the  framework 
established  during  solicitation  planning. 

4.  Source  Selection 

Source  selection  consists  of  applying  the  evaluation  criteria  stated  in  the 
solicitation  to  the  offers  received.  Like  the  previous  step,  source  selection  requires  an 
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adherence  to  the  overall  strategy  selected  in  the  initial  phases  of  the  contract  management 
process.  Depending  on  the  size  and  complexity  of  the  procurement,  this  may  involve 
source  selection  boards,  technical  panels,  and  any  other  expert  required  to  evaluate  the 
offers  received  (Garrett,  2007).  Further,  the  source  selection  may  involve  directly 
negotiating  with  one  or  more  vendors  on  price,  technical  factors,  or  personnel.  Finally, 
the  acquisition  team  selects  the  winner  during  this  step;  it  is  the  most  vulnerable  to 
protests  from  unsuccessful  vendors. 

Due  to  the  visibility  of  source  selections,  senior  officials  have  placed  a  specific 
emphasis  on  the  step,  with  many  of  the  previously  mentioned  refonn  efforts  centered  on 
source  selection.  However,  source  selection  is  actually  an  execution  of  the  evaluation 
strategy.  By  the  time  the  government  has  advanced  to  source  selection,  it  is  typically  too 
late  to  make  significant  changes  without  regressing  back  to  a  previous  step.  Therefore,  it 
is  critical  for  acquisition  teams  to  adequately  perform  the  initial  planning  activities. 

5.  Contract  Administration 

Contract  administration,  like  all  other  steps,  is  vital  to  the  success  of  the  contract 
management  process.  Moreover,  the  length  of  contract  administration  often  dwarfs  each 
of  the  other  steps,  particularly  in  multi-year  contracts.  Successful  contract  administration 
consists  of  adhering  to  procurement  policy  and  enforcing  the  contract  as  written;  often 
complications  occur  when  one  or  both  of  the  parties  fail  to  follow  the  stated  agreement. 
While  change  requests  and  modifications  can  occur  in  any  contract,  the  key  to  effective 
contract  administration  is  a  clear  contract  that  stipulates  the  expectations  of  each  party. 

6.  Contract  Closeout 

The  final  step  in  the  contract  management  process  is  contract  closeout.  This  step 
includes  confirming  that  all  required  work  has  been  accomplished,  as  well  as  any  other 
details  related  to  the  contract.  On  some  occasions,  performance  is  terminated  early;  this 
can  occur  due  to  either  mutual  agreement  or  a  breach  of  contract  by  one  of  the  parties 
(Garrett,  2007).  Clearly,  the  preferred  outcome  is  successful  fulfillment  of  obligations  by 
both  the  buyer  and  the  seller.  However,  when  complications  do  arise,  they  are  often 
caused  by  decisions  made  before  award.  In  fact,  despite  the  abundance  of  guidance 
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concerning  source  selection  and  post-award  actions,  solicitation  planning  remains  “one  of 
the  most  critical  functions  in  contract  management”  (Garrett,  2007,  p.  24).  In  order  to 
understand  the  significance  of  this  step,  one  must  recognize  the  choices  available  to 
acquisition  teams  before  contract  award. 

E.  PRE-AWARD  ACQUISITION  DECISIONS 

One  useful  analogy  for  describing  the  various  alternatives  available  to  contracting 
officers  is  a  decision  tree.  While  dozens  of  contract  types,  procurement  methods,  and 
source  selection  methods  are  available,  decisions  made  early  in  the  process  narrow  the 
field  of  future  possibilities.  For  example,  if  an  acquisition  team  determines  a  product  or 
service  to  be  other  than  commercial,  the  procedures  offered  under  FAR  12  are  no  viable 
procedural  options  (FAR  12.102(a)). 

1.  Contract  Type 

One  of  the  first  decisions  available  to  acquisition  teams  is  the  type  of  contract 
used  to  procure  a  good  or  service.  Covered  in  FAR  16,  contract  type  influences  not  only 
the  procurement  method  and  source  selection  method,  but  the  entire  contract  management 
process  as  well. 

Fixed-price  contracts,  the  first  type,  are  characterized  by  generally  firm  prices. 
The  total  amount  paid  to  the  contractor  is  typically  only  adjusted  to  reflect  incentive  fees, 
economic  adjustments  concerning  component  pricing,  or  the  use  of  a  contract  clause  to 
cover  an  unforeseen  situation  (FAR  16.201).  Since  the  contract  price  is  generally 
determined  before  performance  begins,  the  contractor  shoulders  most  of  the  risk.  If  the 
effort  costs  more  than  originally  planned,  the  contractor  typically  earns  less  profit,  while 
the  government  may  face  little  to  no  consequences. 

In  response,  contractors  typically  factor  this  risk  into  their  proposals,  building  in 
larger  profit  premiums  for  work  that  is  complex  or  prone  to  unknown  hazards.  For  this 
reason,  fixed-price  contracts  are  recommended  for  clearly  defined  goods  and  services, 
such  as  commercial-off-the-shelf  items,  or  efforts  that  a  contractor  has  perfonned  in  the 
past  (FAR  16.104(1)). 
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In  situations  where  the  required  work  is  complex,  undefined,  or  otherwise 
difficult  to  estimate  initially,  acquisition  teams  may  use  the  second  type,  cost- 
reimbursement  contracts.  Cost-reimbursement  contracts  differ  from  fixed-price  contracts 
in  that  the  government  reimburses  the  contractor  for  all  allowable,  reasonable,  and 
allocable  costs  incurred  in  performance  of  the  contract  (FAR  16.301-1).  Since  the 
government  assumes  significantly  more  risk  under  cost-reimbursement  contracts,  the 
profit  premium  afforded  to  contractors  is  typically  lower  under  this  arrangement  (DFARS 
215.404-71-3(c)). 

While  fixed-priced  contracts  are  generally  recommended  for  use  whenever 
practical,  cost-reimbursement  contracts  are  only  allowed  after  an  extensive  review  of  the 
requirement,  the  proposed  contractor,  and  the  contract  administration  team  (FAR  16.301- 
3(a).  Additionally,  acquisition  teams  are  prohibited  from  using  cost-reimbursement 
contracts  to  procure  commercial  items,  limiting  their  use  to  complex,  uniquely 
governmental  efforts. 

2.  Procurement  Method 

After  selecting  the  appropriate  contract  type,  acquisition  teams  must  detennine 
the  best  procurement  method  for  the  acquisition.  The  two  basic  options  differ  primarily  in 
the  evaluation  of  proposals,  yet  the  consequences  of  this  decision  are  reflected  in  nearly 
every  step  of  the  contract  management  process. 

Sealed  bidding,  the  first  method,  consists  of  receiving  vendor  bids  and  selecting  a 
contractor  chiefly  based  on  price,  or  price-related  factors  (Rosa  da  Silva,  2013).  It  is 
relatively  simple,  as  contracting  officers  are  simply  responsible  for  determining  the 
lowest  bid  price  that  conforms  to  the  details  of  the  solicitation.  Sealed  bidding  has  a 
prominent  role  in  the  history  of  federal  contracting;  until  World  War  II,  it  was  the 
preferred  option  for  acquiring  supplies  and  services  (Rosa  da  Silva,  2013).  However, 
sealed  bidding  relies  almost  entirely  on  price-related  criteria,  and  severely  limits 
discretion  in  choosing  a  contract  type;  only  firm-fixed-price  contracts  and  fixed-price 
economic  price  adjustment  contracts  are  permitted  (FAR  14.104).  As  sealed  bidding  only 
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allows  a  narrow  range  of  acquisition  decisions,  it  is  not  suitable  for  many  of  the  complex 
good  and  services  contracting  officers  are  responsible  for  buying  today. 

The  second  method  available  is  contracting  by  negotiation.  This  alternative  was 
first  introduced  as  part  of  the  CICA  under  the  broad  concept  of  best  value,  and  allows 
government  officials  to  discuss  proposal  details  with  offerors  after  receipt  of  proposals. 
Contracting  officers  can  negotiate  in  competitive  procurements  to  determine  the  relative 
merits  of  various  proposals;  or,  in  noncompetitive  instances,  to  discuss  pricing  (Rosa  da 
Silva,  2013).  Based  on  the  wide  discretion  government  officials  have  under  negotiation 
procedures,  each  acquisition  must  have  clearly  stated  evaluation  criteria  so  vendors  know 
how  to  shape  their  proposals.  This  range  of  available  evaluation  options  is  known  as  the 
best  value  continuum,  and  is  explained  in  the  following  section. 

3.  Source  Selection  Method 

In  order  to  understand  the  differences  in  source  selection  methods,  one  must  first 
review  the  best  value  continuum.  Specifically,  it  is  important  to  understand  the  two 
extremes  of  the  continuum  to  assess  the  two  conflicting  goals  of  government  acquisition. 
The  first  goal  of  contracting  officers  is  to  provide  maximum  quality  to  the  end  user. 
Delivery  time,  superior  customer  service,  and  innovative  technical  approaches  are  all 
examples  of  quality.  Regardless  of  the  contract  type,  procurement  method,  or  source 
selection  method  selected  by  the  acquisition  team,  the  contract  will  always  express  some 
level  of  quality  requirements  in  a  contract,  because  an  acquisition  for  insufficient  goods 
or  services  is  never  in  the  best  interest  of  the  government.  On  the  best  value  continuum, 
selecting  the  highest  technically  rated  offeror  represents  a  total  commitment  to  quality 
(FAR  15. 101-l(a». 

However,  as  part  of  instituting  contract  by  negotiation,  the  CICA  also  placed 
restrictions  on  the  method.  Chiefly,  the  act  requires  federal  officials  to  consider  price  in 
every  acquisition,  regardless  of  where  it  falls  on  the  best  value  continuum  (Rumbaugh, 
2010).  While  acquisition  teams  have  the  discretion  to  weigh  price  as  more  or  less 
important  than  other  factors,  the  mandatory  inclusion  of  price  essentially  precludes 
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evaluation  criteria  designed  specifically  to  select  the  highest  technically  rated  offer 
without  regard  to  cost. 

The  second  objective  inherent  in  the  federal  contract  management  process  is  the 
need  to  ensure  the  efficient  use  of  taxpayer  dollars.  Similar  to  quality,  stewardship  of 
public  funds  is  required  for  all  contracts,  and  is  stated  in  the  first  guiding  principle  for  the 
federal  acquisition  system.  Maximum  attention  to  price  as  an  evaluation  factor  is  known 
as  LPTA,  and  it  serves  as  the  opposite  extreme  of  the  best  value  continuum. 

Since  an  evaluation  strategy  aimed  purely  at  selecting  the  highest  technically 
rated  offer  without  regard  to  cost  is  prohibited  by  statute,  the  full  best  value  continuum  is 
technically  not  available  to  acquisition  teams.  Instead,  a  modified  version  of  the  spectrum 
remains  in  its  place,  with  two  main  options:  LPTA  and  tradeoffs.  The  following  is  a  brief 
explanation  of  each  method,  along  with  relative  strengths  and  weaknesses. 

F.  LOWEST  PRICE  TECHNICALLY  ACCEPTABLE 

Under  LPTA  source  selections,  the  government  first  ranks  all  offers  in  reverse 
price  order,  and  then  reviews  them  in  that  order  until  a  technically  acceptable  offer  is 
identified.  If  an  offer  is  found  not  to  be  technically  acceptable,  it  is  eliminated,  and  the 
government  proceeds  to  the  next  lowest  priced  offer.  Under  this  basic  framework, 
acquisition  teams  are  not  authorized  to  accept  a  proposal  other  than  the  technically 
acceptable  low  offer  (FAR,  15.101-2). 

Unsurprisingly,  the  FAR  recommends  LPTA  acquisitions  when  “best  value  is 
expected  to  result  from  selection  of  the  technically  acceptable  proposal  with  the  lowest 
evaluated  price”  (FAR  15.101  -2(a)).  More  specifically,  LPTA  procurements  are  preferred 
when  the  requirement  is  well-defined  and  when  the  govermnent  does  not  anticipate  a 
significant  risk  of  contractor  default  (GAO,  2014).  Since  acquisition  teams  do  not  reward 
vendors  for  proposing  technical  approaches  that  exceed  the  minimum  requirements,  the 
strategy  is  recommended  for  requirements  that  will  not  require  an  innovative  solution. 
Put  another  way,  LPTA  contracts  are  ideal  for  what  private  defense  firm  The  Analytical 

Services  Corporation  (TASC)  describes  as  the  “acquisition  of  commercial  or  non¬ 
complex  services  that  are  clearly  defined  and  expected  to  be  low  risk”  (TASC,  2012,  p. 
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2).  For  example,  the  GAO  highlights  the  government’s  high  usage  of  LPTA  source 
selections  in  commercial  acquisitions  of  fuel  (2014).  Even  for  large  contracts,  fuel  is  a 
relatively  well-understood  requirement  for  buyers  and  sellers,  and  as  long  as  a  vendor  can 
meet  the  basic  contract  terms,  there  is  no  need  to  review  quality  factors  among  the  offers. 

As  a  source  selection  method,  LPTA  satisfies  many  of  the  federal  acquisition 
guiding  principles.  As  contracting  officers  are  required  to  accept  the  lowest  price  offer 
that  meets  the  government  need,  the  customer’s  need  is  expressly  satisfied  in  terms  of 
cost,  while  timeliness  and  performance  are  ostensibly  included  as  well.  Additionally, 
because  LPTA  acquisitions  tend  to  be  simpler  than  tradeoffs,  contracting  offices  can 
move  more  quickly  through  the  six-step  contract  management  process,  reducing 
administrative  operating  costs.  Finally,  the  generally  inflexible  nature  of  the  LPTA 
source  selection  method  does  not  grant  contracting  officers  discretion,  which  serves  as  a 
guard  against  the  appearance  of  favoritism,  promoting  the  perception  of  integrity, 
fairness,  and  openness. 

G.  TRADEOFF 

By  contrast,  tradeoff  acquisitions  allow  contracting  officers  the  flexibility  to 
weigh  non-price  factors  as  part  of  an  overall  evaluation  assessment.  Under  this  method, 
the  acquisition  teams  have  the  flexibility  to  select  other  than  the  LPTA  offer  if  the 
perceived  benefits  of  a  higher  priced  offer  outweigh  the  additional  cost  (FAR  15.101- 
1(c)).  This  decision  must  confonn  to  the  evaluation  criteria  stated  in  the  solicitation; 
however,  the  mere  ability  to  weigh  multiple  components  of  an  offer  can  be  useful  in 
complex,  technical  acquisitions. 

As  the  GAO  (2014)  notes,  tradeoffs  are  valuable  when  the  past  performance  or 
technical  capacity  of  a  contractor  is  important;  goods  or  services  that  require  a 
contractor’s  experience  throughout  the  acquisition  process  may  warrant  the  use  of  the 
tradeoff  method.  These  are  most  commonly  found  in  complex  acquisitions  where  the 
government  anticipates  unknown  challenges  in  the  future,  that  is,  a  specific  need  cannot 
be  identified  at  the  time  of  award,  but  the  increased  capability  of  qualified  contractors 
may  be  beneficial.  This  dialogue  of  evaluation  factors  and  their  relative  importance  is 
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known  as  an  integrated  assessment,  and  is  essentially  “the  crux  of  the  entire  source 
selection”  (Slate,  2004,  p.  38). 

Compared  to  the  LPT  A  method,  tradeoffs  emphasize  providing  maximum 
performance  for  the  customer,  although  costs  may  increase  as  a  result.  However, 
acquisition  teams  rely  on  price  competition  among  offerors  to  ensure  fair  and  reasonable 
offers  (FAR  15.402(a)(2)).  Additionally,  proponents  of  tradeoffs  argue  that  the  initial 
costs  of  a  higher  priced  vendor  are  ultimately  more  efficient,  as  the  incentive  structure 
encourages  vendors  to  avoid  cutting  costs  that  could  jeopardize  the  effort  after  award.  By 
this  logic,  the  resulting  contract  is  less  likely  to  require  modifications  after  award  due  to 
the  contractor’s  increased  capability.  Finally,  price  competition  is  still  typically  present. 

In  terms  of  fairness  and  openness,  tradeoffs  certainly  expose  the  govermnent  to 
the  perception  of  internal  bias.  While  tradeoffs  are  specifically  structured  to  reduce  this 
risk — for  example,  acquisition  teams  are  required  to  state  the  relative  importance  of  price 
versus  non-price  factors — the  strategy  ultimately  carries  the  perception  of  less 
transparency. 

H.  IMPLICATIONS  OF  EACH  SOURCE  SELECTION  METHOD 

While  in  some  cases  the  choice  between  the  LPT  A  and  tradeoff  source  selection 
methods  is  clear,  the  majority  of  acquisitions  require  government  teams  to  analyze  the 
situation  and  weigh  the  relative  benefits  of  each  method  against  their  costs.  To 
understand  this  process,  we  have  summarized  the  key  implications  of  each  method. 

However,  it  is  important  to  note  that  these  conclusions  are  not  necessarily 
representative  of  the  entire  population  of  federal  contracts.  Many  of  the  findings  come 
from  GAO  reports,  which,  while  informative,  are  typically  based  on  small  sample  sizes 
(15  to  20  contracts).  In  other  cases,  the  conclusions  are  drawn  from  academic  journals  or 
sentiments  of  acquisition  personnel  (GAO,  2014),  which  also  lack  the  statistical 
significance  of  a  data  analysis. 
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1. 


LPTA 


The  most  significant  weakness  inherent  in  LPTA  procurements  is  the  potential  for 
unrealistic  pricing.  The  reliance  on  price  as  the  final  discriminator  in  LPTA  source 
selections  has  a  negative  tendency  to  encourage  a  race  to  the  bottom  by  competing 
vendors.  The  Department  of  State’s  (DOS’s)  contractor,  Annor  Group  North  America 
(AGNA),  provides  an  excellent  case  study  for  this  behavior,  as  the  contractor’s 
shortcomings  were  directly  related  to  their  unsustainable  pricing  structure  (CWC,  2009). 
Rather  than  inexperienced  management  or  unethical  employees,  the  majority  of 
contractor  violations  resulted  from  cost-cutting  measures:  lack  of  security  vehicles,  third 
country  national  personnel  incapable  of  communicating  in  English,  and  shortage  of 
employee  uniforms  (CWC,  2009).  The  AGNA  example  is  particularly  troubling  because 
the  DOS  was  not  allowed  to  utilize  tradeoffs  for  overseas  security  acquisitions  at  the 
time,  requiring  the  use  of  an  inappropriate  source  selection  method. 

In  order  to  combat  this,  acquisition  teams  often  employ  cost  realism:  a  methodical 
review  of  proposed  pricing  against  the  required  perfonnance  to  identify  and  respond  to 
unrealistically  low  offers.  Cost  realism  was  originally  designed  as  a  technique  to  mitigate 
unrealistic  pricing  on  cost-reimbursement  contracts,  where  the  potential  risk  to  the 
government  is  the  greatest,  but  contracting  officers  can  also  use  cost  realism  language  in 
fixed  price  contracts  in  to  achieve  the  same  result  (Lohfeld,  2014).  However,  this  solution 
must  be  pre-meditated;  evaluating  offers  for  cost  realism  without  specifically  stating  so  in 
the  solicitation  represents  a  departure  from  the  source  selection  criteria,  and  a  “fertile  area 
for  procurement  protest”  (Lohfeld,  2014). 

Despite  efforts  to  promote  cost  realism,  the  perceived  threat  to  offerors  of 
unrealistically  low  pricing  by  competitors  is  often  enough  to  dissuade  a  bid,  because  the 
resources  spent  creating  an  offer  appear  to  be  a  waste  (CWC,  2009).  Regardless  of 
whether  the  government  would  have  chosen  these  technically  superior  contractors,  their 
presence  increases  the  competitive  nature  of  the  acquisition,  increasing  government 
value.  Removing  them  from  the  acquisition  is  not  good  for  the  government. 
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In  other  cases,  the  need  to  remain  competitive  may  persuade  firms  to  minimize 
their  staff  and  research  funding  in  order  to  reduce  operating  costs.  As  noted  by  TASC 
(2012),  general  and  administrative  (G&A)  costs  are  typically  the  business  expense 
eliminated,  followed  swiftly  by  research  and  development  (R&D)  funding.  While  large 
firms  such  as  Lockheed  Martin  and  Northrup  Grumman  typically  have  several  projects  to 
move  personnel  around,  small  firms  may  have  to  radically  alter  their  entire  operation  to 
win  bids,  permanently  reducing  their  potential  technical  innovation. 

Further,  among  contractors  who  maintain  their  original  staffing  levels,  there  is  no 
incentive  to  present  a  high  quality  product  or  service  (Nichols  &  Totman,  2013).  In  fact, 
LPT  A  procurements  actually  discourage  any  level  of  effort  beyond  the  stated 
minimum — quoting  additional  resources  simply  drives  up  the  proposed  cost  without 
increasing  the  attractiveness  of  an  offer.  In  a  recent  National  Contract  Management 
Association  (NCMA)  presentation,  the  organization  advised  vendors  to  bid  for  the 
absolute  minimum  level  of  work  required  and  to  cut  supervisory  personnel  on  LPTA 
projects,  even  though  the  strategy  “could  be  deemed  risky  to  some  companies”  (Calhoon 
&  Sybert,  2012). 

2.  Tradeoffs 

Conversely,  tradeoffs  often  suffer  from  the  perception  of  increased  costs. 
Ultimately,  trading  higher  prices  in  exchange  for  a  more  robust  technical  proposal  or 
superior  past  performance  means  paying  some  fonn  of  performance  premium,  as  part  of 
the  integrated  assessment  process.  While  the  acquisition  team  may  be  fully  justified  in 
paying  a  higher  cost,  outside  observers — both  public  and  private — may  disagree.  The 
GAO  noted  that,  among  best-value  acquisitions,  the  percentage  of  tradeoffs  declined 
steadily  from  2009  to  2013,  the  period  for  which  they  analyzed  the  metric  (2014). 
Additionally,  DOD  contracting  officers  and  program  managers  echoed  these  findings, 
attributing  the  heightened  use  of  LPTA  to  shrinking  budgets  caused  by  sequestration. 
From  the  industry  perspective,  this  opinion  is  repeated;  TASC  has  attributed  the  decline 
in  tradeoffs  to  shrinking  budgets  (TASC,  2012).  While  the  government  is  required  to 
evaluate  price  reasonableness  for  all  offers,  preferably  through  the  use  competitive 
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proposals  (FAR  15.305(a)(1)),  the  increased  scrutiny  brought  on  by  the  tradeoff  method 
appears  to  influence  source  selection  method  nevertheless. 

Additionally,  the  increased  flexibility  offered  by  tradeoff  requires  a  pool  of 
adequately  trained  personnel  to  successfully  conduct  the  source  selection.  Unlike  LPTA 
source  selection,  which  only  assesses  the  threshold  of  acceptability,  acquisition  teams 
must  define  and  quantify  each  evaluation  factor  as  part  of  each  procurement’s  specific 
integrated  assessment.  The  GAO  noted  that  “developing  non-cost  factors  that 
meaningfully  discriminate  between  offers”  can  be  particularly  difficult  (2010,  p.  17). 
Their  report  also  highlights  the  need  to  bring  in  personnel  with  specific  expertise,  citing  a 
U.S.  Army  Corps  of  Engineers  project  where  water  flow  experts  were  brought  in  to 
analyze  the  merits  of  different  proposals  (GAO,  2014). 

Moreover,  the  requisite  training  cannot  be  accomplished  by  a  new  DAU  offering. 
In  interviews  with  the  GAO,  DOD  officials  stressed  the  importance  of  on-the-job  training 
for  acquisition  personnel,  specifically  when  deciding  between  the  two  source  selection 
methods,  as  well  as  justifying  the  decision  to  select  a  higher  priced  offer  (GAO,  2014). 
This  is  particularly  relevant  today  because  the  DOD  increased  the  size  of  the  contracting 
workforce  in  the  years  leading  up  to  fiscal  year  2015.  Training  these  new  personnel  in  the 
nuances  of  tradeoffs  will  likely  take  decades. 

Another  consequence  of  the  complicated  nature  of  tradeoffs  is  the  increased  risk 
of  protests  from  losing  offerors.  Within  federal  acquisition,  two  factors  significantly 
increase  the  risk  of  protest:  dollar  value  and  agency  discretion.  As  Carl  Gebo,  a  partner  at 
Adomo  &  Yoss  (2009),  explains, 

In  negotiated  procurements,  the  interaction  between  offerors  and  the  agency 
can  be  far  greater  than  in  sealed  bid  procurements.  In  seeking  clarifications, 
conducting  discussions,  and  making  presentations,  suspicions  of 
prejudicially  different  treatment  thrive  and  add  to  the  incentive  to  protest. 

The  opportunity  to  influence  those  evaluations  through  those  agency 
contacts  is  real  and  even  part  of  the  process.  When  looked  at  in  retrospect, 
through  the  prism  of  a  lost  contract,  offerors  are  inclined  to  believe  that 
undue  influence  or  unfair  treatment  occurred,  (p.  30) 
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Based  on  the  government’s  increased  use  of  the  tradeoff  method  for  high-dollar 
acquisitions,  particularly  above  $25  million,  as  well  as  the  subjective  nature  of 
conducting  an  integrated  assessment,  the  risk  of  protest(s)  is  much  higher  for  these 
procurements. 

Even  among  highly  trained  personnel  who  manage  to  avoid  protests,  tradeoffs  can 
often  be  lengthy  affairs,  especially  when  compared  to  LPTA  procurements.  Developing 
meaningful  criteria,  effectively  stating  the  evaluation  methodology,  and  conducting  the 
integrated  assessment  are  all  deliberate  processes,  resulting  in  longer  procurements. 
During  a  GAO  study,  acquisition  personnel  echoed  the  measured  pace  of  the  tradeoff 
method,  citing  a  typical  timeline  of  18  to  24  months  for  completion  (2010).  While  the 
GAO  based  its  findings  on  individual  interviews  rather  than  contract  data  collection,  the 
point  remains.  This  sentiment  is  echoed  by  industry,  which  discredits  the  LPTA  method 
but  grudgingly  admits  its  streamlined  efficiency  (Lohfeld,  2015). 

The  final,  and  perhaps  most  troubling,  drawback  of  the  tradeoff  method  is  the 
government’s  apparent  lack  of  actually  trading  off  price  and  non-price  factors.  In  an 
FY2009  study,  the  GAO  found  that  DOD  contracting  officers  utilized  the  tradeoff 
method  in  68%  of  their  sample  (2010).  However,  of  these,  the  government  selected  a 
higher  priced  offer  less  than  a  quarter  of  the  time.  Even  more  troubling,  selecting  the  non- 
low  offer  resulted  in  a  protest  rate  of  72%,  with  nearly  a  third  of  protests  resulting  in 
either  acquisition  cancellation  or  changed  award  decision  (GAO,  2010).  Put  simply,  the 
government  seems  unwilling  or  unable  to  utilize  the  principal  benefit  of  the  tradeoff 
method.  Given  the  increased  time  and  training  required,  as  well  the  increased  risk  of 
protests,  the  illusion  of  choice  may  not  be  in  the  government’s  best  interest. 

I.  SUMMARY 

In  this  chapter,  we  have  reviewed  the  context  for  the  tradeoff  source  selection 
method.  Prior  to  the  CICA,  the  government  typically  utilized  some  version  of  LPTA, 
with  a  primary  focus  on  acquiring  goods  and  services  at  the  lowest  price.  As  government 
regulation  evolved,  so  did  the  guiding  principles  of  acquisition,  resulting  in  objectives 
other  than  cost  savings.  The  federal  government  introduced  tradeoffs  along  with  the  best 
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value  continuum  as  a  way  to  encourage  innovative  solutions  for  complex  procurement 
problems.  Based  on  the  relative  strengths  and  weaknesses  of  the  tradeoff  method,  it  is 
typically  advised  for  complex,  developmental,  less-defined  procurements. 

However,  as  discussed  previously,  many  of  these  conclusions  are  drawn  from 
incredibly  small  sample  sizes,  while  others  simply  express  personal  experiences.  In  the 
next  chapter,  we  explain  the  methodology  of  our  study  in  order  to  test  the  validity  of 
these  claims  using  contract  data. 
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III.  METHODOLOGY 


A.  INTRODUCTION 

In  this  section,  we  examine  our  process  for  collecting  and  analyzing  relevant  data 
to  answer  the  research  questions  of  this  study.  We  hope  that  our  approach  can  pave  the 
way  for  future  studies  of  this  subject  across  all  military  branches.  This  section  will 
provide  a  brief  examination  of  the  contracting  structure  and  systems  used  in  the  Air 
Force.  The  subsequent  sections  explain  the  data  collection  method  and  the  unique 
elements  of  our  research. 

B.  CONTRACTING  STRUCTURE  AND  SYSTEMS 

This  section  provides  an  overview  of  the  structure  and  systems  used  in  the  Air 
Force  to  carry  out  the  contracting  mission. 

1.  Base  Operational  Contracting 

Within  the  Air  Force,  most  installations  include  a  contracting  contingent 
(typically  a  fully  staffed  squadron)  responsible  for  procurement  needs  across  the  base. 
Contracting  officers  (COs)  and  contracting  specialists  within  the  squadron  are  responsible 
for  procurement  planning,  solicitation  planning,  solicitation,  source  selection,  contract 
administration,  and  contract  closeout  for  every  acquisition  (Garret,  2007).  The  discharge 
of  these  duties  involves  ensuring  documentation  of  such  actions  is  complete  within  the 
contract  file  and  all  associated  databases. 

2.  Federal  Procurement  Data  System-Next  Generation 

The  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG)  provides  the 
platfonn  for  COs  to  report  contract  data  to  government  officials.  Within  this  system,  each 
contract  action  has  a  corresponding  Contract  Action  Report  (CAR)  that  captures  a 
significant  amount  of  data  relating  to  the  contract,  including  dollar  amount,  length  of 
perfonnance,  number  of  offers  received,  and  the  extent  to  which  the  purchase  was 
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competed.  These  data  are  unique  to  each  contract  action  and  are  submitted  and 
maintained  in  the  FPDS-NG  system. 

3.  Federal  Business  Opportunities 

Another  valuable  source  of  contract  information  is  the  government-wide  point  of 
entry,  otherwise  known  as  the  Federal  Business  Opportunities  (FBO)  website 
(www.fbo.gov).  Contracting  squadrons  upload  solicitation  documents  to  the  website  to 
infonn  potential  contractors  of  govermnent  requirements.  The  solicitation,  maintained  on 
the  FBO  website,  contains  proposal  preparation  and  submission  instructions,  states  the 
evaluation  factors  that  the  source  selection  team  will  use  to  assess  each  proposal,  and 
identifies  which  source  selection  method  will  be  used  to  make  the  final  award  decision. 
The  FBO  database  stores  solicitations  and  all  related  amendments,  as  well  as  additional 
infonnation  associated  with  the  requirement  (e.g.,  drawings  or  specifications,  answers  to 
questions  posed  by  potential  vendors,  etc.). 

4.  Contractor  Performance  Assessment  Reporting  System 

The  Air  Force  and  other  government  agencies  use  the  Contractor  Performance 
Assessment  Reporting  System  (CPARS)  to  report  their  periodic  evaluations  of  contractor 
adherence  to  contract  terms.  This  system  stores  contractor  perfonnance  evaluations  and 
supporting  rationale  for  each  contract.  FAR  Subpart  42.15  provides  instructions  for 
evaluating  contractors  based  on  the  elements  of  cost  control,  technical,  schedule,  business 
relations,  and  subcontracting.  Within  these  broad  categories,  the  contractor  is  given  an 
evaluation  according  to  a  Likert-style  scale,  which  is  discussed  further  in  the  next 
chapter.  As  demonstrated  in  the  following  sections  and  chapters,  these  ratings  serve  as  a 
proxy  rating  of  overall  contract  performance;  higher  ratings  indicate  compliance  with 
contract  requirements  and  positive  perfonnance. 

C.  DATA  COLLECTION 

Although  the  aforementioned  systems  contain  data  relevant  to  our  research,  we 
needed  access  to  more  in-depth  data  that  is  not  available  in  any  system  or  database.  As 
such,  we  traveled  to  two  Air  Force  contracting  squadrons  to  review  their  contract  files 
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and  extract  the  necessary  data.  Specifically,  we  captured  contract  outcome  data  from 
contracts  that  used  the  tradeoff  source  selection  method  and  compared  that  information 
against  the  same  data  points  gathered  from  contracts  using  the  LPTA  source  selection 
method. 

We  focused  our  research  on  contracts  awarded  by  the  Air  Force’s  21st 
Contracting  Squadron  located  at  Peterson  Air  Force  Base,  and  the  50th  Contracting 
Squadron  located  at  Schriever  Air  Force  Base.  We  felt  that  the  Colorado  Springs  area, 
where  these  bases  are  located,  offered  significant  potential  for  gathering  large  amounts  of 
data,  due  to  the  close  proximity  of  the  two  bases.  Also,  Peterson  Air  Force  Base  is  the 
headquarters  of  Air  Force  Space  Command,  and  we  knew  that  it  would  provide  a 
sufficient  mix  of  contracts  likely  to  involve  both  tradeoff  and  LPTA  procedures.  With  the 
help  of  the  Air  Force  Installation  Contracting  Agency  (AFICA),  we  contacted  both  bases 
to  request  that  they  identify  contracts  with  the  following  criteria: 

•  available  CPARS  data 

•  high  dollar  amount,  preferably  valued  above  $700,000 

•  an  equal  number  of  contracts  that  utilized  tradeoff  and  LPTA  source 
selection  methods 

By  coordinating  with  each  point  of  contact,  a  sufficient  number  of  contracts  were 
made  available  to  us  upon  our  arrival  at  each  base.  Between  the  two  installations,  we 
accessed  data  from  33  contract  files  dealing  with  a  broad  range  of  requirements. 

The  five  categories  of  data  that  we  were  interested  in  locating  for  our  study  were: 

•  basic  contract  information 

•  acquisition  complexity 

•  enviromnental  factors 

•  outcome  variables 

•  other  data  (relating  to  discussions) 

Basic  contract  information  included  the  main  identifying  features  of  a  contract, 
such  as  the  contract  number  and  description,  as  well  as  the  buying  organization 
(squadron).  The  North  American  Industry  Classification  System  (NAICS)  code  and 
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Product  Service  Code  (PSC)  were  also  captured  as  further  identifying  characteristics  for 
each  contract. 

Acquisition  complexity  was  measured  using  such  factors  as  contract  type,  set- 
asides  used,  length  of  contract,  number  of  contract  line  items,  and  dollar  value.  In  this 
section,  we  note  the  source  selection  method  utilized,  as  well  as  the  number  and 
description  of  evaluation  factors  and  subfactors.  Industry  involvement  was  measured  by 
whether  or  not  conferences  were  held  in  relation  to  solicitation  and  communication  of 
requirements,  how  many  offerors  submitted  proposals,  and  whether  there  were  multiple 
awards  or  a  single  contract  award.  Incentive  and  award  fee  criteria,  where  present,  were 
also  relevant  to  the  acquisition  complexity  measure. 

Environmental  factors  included  the  number  of  reviews,  as  well  as  the  utilization 
of  Multi-Functional  Independent  Review  Team  support.  Here  we  also  gave  each  contract 
file  a  rating  for  level  of  documentation  on  a  five-point  scale  where  I  =wcll  below  average, 
2=below  average,  3=average,  4=above  average,  and  5=well  above  average.  We  also 
included  an  assessment  of  the  state  of  the  U.S.  economy  at  the  time  of  contract  award,  in 
order  to  assess  the  larger  economic  context  of  each  acquisition. 

Outcome  variables  were  crucial  to  our  study  because  they  provided  the  dependent 
variables  and  basis  for  comparison  of  the  two  source  selection  methods.  The  PALT  and 
CPARS  ratings  (further  expounded  upon  in  the  next  chapter)  were  the  main  factors  used 
to  evaluate  contract  outcomes.  Other  important  measures  of  contract  outcome  included 
protest  data,  Earned  Value  Management  (EVM)  data,  number  of  solicitation 
amendments,  and  earned  award  fee. 

The  last  category  of  information,  other  data,  included  discussions  between  the 
government  and  vendors  during  source  elections.  For  example,  we  looked  at  requests 
from  the  contracting  office  to  offerors  for  clarifications  of  their  proposals.  When  the 
source  selection  team  notices  a  deficiency  in  an  offeror’s  proposal,  a  deficiency  report  or 
an  Evaluation  Notice  (EN)  is  issued  to  make  the  problem  known.  We  counted  the  number 
of  ENs,  as  well  as  the  number  of  rounds  of  ENs,  to  better  understand  the  quality  of 
proposals  as  well  as  the  effectiveness  of  the  source  selection.  The  final  data  point  we 
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collected  was  the  basis  for  award;  if  the  contract  was  awarded  on  any  basis  other  than 
cost  or  price,  we  noted  the  determining  factor. 

D.  UNIQUE  ELEMENTS  OF  OUR  RESEARCH  METHOD 

By  working  with  our  contacts  at  each  base  prior  to  our  travel,  we  ensured  that 
there  would  be  a  sufficient  and  equal  number  of  contracts  that  utilized  the  two  different 
source  selection  methods.  If  we  had  accessed  a  list  of  each  base’s  contracts  and  requested 
specific  contracts  be  made  available,  we  might  have  encountered  the  problem  of  having 
too  many  sample  cases  of  one  source  selection  method  and  not  enough  of  the  other.  Our 
contacts’  prior  knowledge  of  each  contract  helped  us  avoid  that  problem.  By  requesting 
their  assistance,  we  also  avoided  the  potential  pitfall  of  pre-selecting  contracts  for 
evaluation  only  to  find  upon  arrival  at  the  base  that  those  contracts  had  been  closed  and 
staged. 

Another  positive  element  of  this  approach,  albeit  intangible,  was  that  it  involved 
other  contracting  specialists  and  contracting  officers  who  would  otherwise  not  have  taken 
part  in  the  study.  Sharing  responsibility  for  this  research  helped  to  disseminate  the 
thought  process  and  goals  of  this  research  down  to  the  squadron  level.  By  involving 
others,  they  are  now  aware  that  AFICA  is  engaged  in  research  that  may  streamline  the 
procurement  process  and  improve  contracting  operations.  As  contracting  personnel 
communicate  and  transfer  across  squadrons,  other  installations  will  have  an  increased 
awareness  of  the  data  needed  for  future  studies  associated  with  this  research. 

One  unfortunate  finding  was  the  lack  of  EVM  data.  Major  defense  systems 
acquisition  programs  would  be  more  likely  to  employ  EVM  practices.  Although  Peterson 
AFB  and  Schriever  AFB  both  managed  high  dollar  contracts  dealing  with  space  systems, 
they  dealt  mainly  with  system  support  rather  than  system  development;  therefore,  the 
level  of  complexity  did  not  necessitate  EVM  processes.  Thus,  we  were  unable  to  capture 
EVM  data  from  the  contracts  made  available  to  us. 

Another  challenge  of  our  research  method  was  the  sheer  magnitude  of  the  contract 
files  to  be  examined.  Some  files  were  unexpectedly  large,  with  portions  stored  in 
different  locations  within  the  office  building.  Several  times,  a  member  of  our  group 
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would  begin  examining  the  pre-award  data  for  a  contract  only  to  subsequently  discover 
that  the  award  document  was  stored  elsewhere.  Each  installation’s  security  measures 
were  unique,  and  this  caused  some  delays  in  finding  crucial  documents.  This  was 
particularly  the  case  at  Schriever  AFB.  While  this  issue  did  not  significantly  impede  our 
progress,  it  did  slow  the  process  and  constrain  our  already  limited  time  on  site. 

E.  SUMMARY 

Overall,  our  data  collection  methodology  accomplished  the  end  goal  of  gathering 
a  sufficient  amount  of  data  to  examine  our  research  questions.  Colorado  Springs  provided 
an  excellent  location  for  our  research;  we  were  able  to  examine  the  contracts  of  two 
separate  squadrons,  which  provided  us  with  a  mix  of  LPTA  and  tradeoff  contracts  and 
their  corresponding  outcomes.  The  next  chapter  presents  the  results  of  our  data  analysis. 
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IV.  RESULTS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

Once  we  collected  the  data  mentioned  in  the  previous  chapter,  we  used  statistical 
software  to  perform  a  series  of  analyses.  In  the  following  sections,  we  discuss  the 
assumptions  tested  before  we  ran  the  program,  followed  by  the  results  of  the  data  analysis 
and  conclusions  to  be  drawn  therefrom. 

B.  DATA  DESCRIPTION 

For  this  analysis,  we  used  two  main  outcomes,  or  dependent  variables  (DVs): 
procurement  administrative  lead  time  (PALT)  and  Contractor  Perfonnance  Assessment 
Reporting  System  (CPARS)  data.  PALT  assesses  time  to  contract  by  calculating  the 
number  of  days  between  receipt  of  the  requisition  and  contract  award.  It  is  a  continuous 
variable.  CPARS  serves  as  a  proxy  measure  of  contract  success  or  failure  by  using  the 
ratings  given  to  each  completed  contract.  CPARS  ratings  are  given  in  Likert-style 
responses  where  1=  unsatisfactory,  2=marginal,  3=satisfactory,  4=very  good,  and 
5=excellent.  In  this  case,  we  used  only  combined  CPARS  scores  as  the  second  dependent 
variable.  Specifically,  we  searched  for  differences  in  CPARS  ratings  between  LPTA  and 
tradeoff  source  selections  by  examining  the  average  CPARS  rating  for  each  contract.  The 
average  rating  was  composed  of  the  most  frequently  recorded  CPARS  scores:  CPARS 
quality  ratings,  CPARS  schedule  ratings,  and  CPARS  business  relationship  ratings. 

Our  model  has  one  independent  variable  (IV).  Independent  variables  are  those 
that  can  be  manipulated  by  the  researcher  (or  user)  to  evoke  a  change  in  the  outcome,  or 
DV.  In  this  case,  the  IV  concerns  the  source  selection  methodology  used  for  the  contract: 
LPTA  or  tradeoff.  Source  selection  methodology  is  a  choice  made  by  the  “user”  (the 
integrated  product  team,  which  includes  the  contracting  officer);  hence  it  is  considered  an 
IV.  Our  IV  is  labeled  LPTATO  and  it  is  a  binary  variable  where  0=LPTA  and  l=tradeoff. 

Finally,  there  are  three  covariate  variables.  Covariates  are  secondary  variables  that 
can  also  affect  the  relationship  of  primary  interest:  the  relationship  between  the  IV  and 
the  DV.  Specifically,  covariates  are  variables  other  than  the  independent  variable  that 
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potentially  affect  the  outcome  variable,  or  DV.  In  this  case,  the  covariates  are  (1)  number 
of  evaluation  factors  (NUMEVALF ACTORS),  (2)  number  of  reviews  (NUMREVIEWS), 
and  (3)  number  of  offers  (NUMOFFERS).  All  are  continuous  variables.  Theoretically, 
each  of  these  covariates  could  affect  the  PALT  positively  (e.g.,  a  larger  number  of  any  of 
the  covariates  could  increase  the  PALT,  or  time  to  contract).  In  this  case,  we  hope  to 
parcel  out  the  effects  of  the  covariates  in  order  to  more  clearly  see  the  effect  the 
contracting  methodology  (LPTATO)  has  on  the  outcome  variables  (PALT  and  CPARS 
ratings).  It  is  worth  noting  that  we  wanted  to  use  contract  dollar  value  (VALUE)  as  a 
covariate,  however  VALUE  did  not  pass  the  homogeneity  of  regression  test  (see 
paragraph  4  below),  therefore  we  could  not  use  it  in  the  analysis. 

1.  Descriptive  Statistics 

Basic  descriptive  statistics  for  each  variable  are  shown  in  Table  1.  The  table 
presents  three  figures  for  each  variable:  (1)  the  total  for  all  the  data,  (2)  the  total  for 
LPTA  contracts,  and  (3)  the  total  for  tradeoff  contracts. 

2.  Data  Limitations 

The  data  point  NUMOFFERS,  which  proved  to  be  an  important  variable,  was 
collected  from  the  Contract  Action  Report  (CAR)  on  the  Federal  Procurement  Data 
System — Next  Generation  (FPDS-NG).  Taking  this  number  from  the  CAR  was  more 
efficient  and  practical  than  counting  each  offer  by  hand;  counting  each  offer  ourselves 
would  have  proven  to  be  an  unwise  use  of  our  limited  time  available  for  data  collection. 
The  limitation  with  relying  on  the  CAR,  however,  is  that  we  became  subject  to  human 
error  that  could  have  occurred  at  the  time  of  initial  entry  into  FPDS-NG.  Infonnation 
found  in  the  CAR  is  only  as  good  as  the  care  taken  to  input  the  data.  Using  the  CAR  to 
count  NUMOFFERS  was  the  best  approach;  however,  possible  human  error  presents  a 
limitation. 

The  more  substantial  limitation  of  our  data  is  its  sample  size  and  distribution. 
With  only  33  cases,  the  sample  size  is  small  to  moderate;  however,  the  distribution  of 
cases  by  source  selection  method  is  unbalanced.  Power  calculations  suggest  the  need  for 
14  cases  for  each  method  (i.e.,  14  LPTA  cases  and  14  tradeoff  cases)  in  order  to  achieve 
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adequate  power  (a  =  .05,  (B  =  .80).  The  data  are  unbalanced  with  respect  to  the  number  of 
cases  for  each  contracting  methodology.  There  are  10  LPTA  cases  and  23  tradeoff  cases. 
This  unbalanced  design  can  cause  ambiguity  about  the  mean  as  the  intercept  and  make 
assignment  of  sums  of  squares  more  difficult.  There  are,  however,  solutions  to  these 
issues.  A  weighted  mean  can  be  used  in  place  of  the  grand  mean1  and  the  STATA 
software  automatically  handles  the  assignment  of  the  sums  of  squares.  Thus,  we 
proceeded  with  our  analysis  despite  these  limitations. 


Table  1.  Descriptive  Statistics 


Variable 

Obs 

Mean 

StdDev 

Min 

Max 

PALT 

(days) 

33 

352.7879 

334.3218 

21 

1019 

10 

216.4 

198.8524 

30 

615 

23 

412.087 

366.4262 

21 

1019 

Average 

CPARS 

Rating 

22 

4.256061 

.7717661 

3 

5 

6 

4.302778 

.7895439 

3 

5 

16 

4.238542 

. 7904925 

3 

5 

Number  of 
Evaluation 
Factors 

29 

2.344828 

.6695341 

1 

4 

10 

2 

.4714045 

1 

3 

19 

2.526316 

.6966923 

2 

4 

Number  of 
Reviews 

26 

9 

6.734983 

3 

28 

10 

9.6 

6.328068 

3 

25 

16 

8.625 

7.154253 

3 

28 

Number  of 
Offers 

33 

5.666667 

5.677074 

2 

22 

10 

3.2 

1.549193 

2 

6 

23 

6.73913 

6.475249 

2 

22 

Bold  =  total  for  all  data,  non-italicized  =  LPTA,  italicized  =  tradeoff 


3.  Analysis 

Because  our  intent  is  to  analyze  differences  in  contract  outcomes  based  on  source 
selection  methods,  a  group  comparison  statistical  methodology  is  necessary.  In  other 
words,  the  source  selection  methods  are  divided  into  two  groups,  LPTA  and  tradeoff,  and 
we  seek  to  find  if  there  are  differences  in  contract  outcomes  between  each  group. 


1  The  grand  mean  would  be  the  intercept  in  a  balanced  design. 
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We  used  a  technique  called  multivariate  analysis  of  covariance,  or  MANCOVA, 
to  assess  group  differences.  MANCOVA  essentially  creates  a  new  dependent  variable 
using  the  information  from  the  given  DVs  (PALT  and  CPARS  ratings).  This  new 
dependent  variable  is  created  in  a  way  that  maximizes  differences  between  the  grouping 
variable  (LPTA  or  tradeoff  source  selections).  MANCOVA  addresses  the  following 
questions:  Are  mean  differences  among  the  groups  on  a  combination  of  DVs  (after 
adjusting  for  covariate  effects)  likely  to  have  occurred  by  chance?  Taken  from  another 
angle,  is  there  a  significant  difference  between  the  mean  value  for  PALT  and  CPARS 
ratings  in  the  LPTA  acquisitions  versus  the  mean  value  for  PALT  and  CPARS  ratings  in 
the  tradeoff  acquisitions,  once  the  effect  of  the  covariates  (NUMEVALF ACTORS, 
NUMREVIEWS,  NUMOFFERS)  have  been  parceled  out?  MANCOVA  examines  the 
relationships  between  the  DVs  (PALT  and  CPARS  ratings)  and  the  independent  variable 
(choice  of  LPTA  or  tradeoff  methodology)  while  taking  into  account  the  effect  the 
covariates  might  have  on  the  DVs. 

If  differences  in  outcomes  are  found  using  MANCOVA,  researchers  typically 
choose  to  dig  deeper  into  the  differences  using  a  univariate  technique  called  analysis  of 
covariance  (ANCOVA).  ANCOVA  essentially  performs  the  same  “group  differences” 
analysis  as  a  MANCOVA;  however,  because  it  is  univariate  in  nature,  this  method 
assesses  one  DV  at  a  time  (PALT  or  CPARS,  rather  than  PALT  and  CPARS).  This 
method  helps  researchers  isolate  where  the  difference  is  occurring  (which  IVs  or 
covariates  are  affecting  which  DVs). 

4.  Assumption  Testing 

Before  conducting  the  MANCOVA,  certain  assumptions  about  the  data  were 
tested.  First,  we  assessed  multivariate  normality  by  examining  density  graphs, 
determining  multivariate  skewness  and  kurtosis,  and  examining  the  Doomik-Hansen  test 
for  multivariate  normality  (Doornik  &  Hansen,  2008).  For  the  DVs,  both  PALT  and 
CPARS  were  deemed  to  be  non-normal.  Both  variables  were  normalized  via  a 
logarithmic  transformation.  All  of  the  covariates  (NUMEVALFACTORS, 
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NUMRE VIEWS,  NUMOFFERS)  also  required  a  logarithmic  transformation,  after  which 
they  were  deemed  to  be  multivariate  nonnal. 

Second,  we  searched  for  outliers  using  Mahalanobis’  distance,  and  found  no 
significantly  influential  outliers  in  our  data.  Third,  we  assessed  linearity  by  examining 
scatter  plots  of  (1)  the  paired  DVs,  (2)  all  pairs  of  covariates,  and  (3)  all  pairs  of  DV- 
covariate  combinations  for  each  source  selection  method  (LPTA  and  tradeoff,  a  total  of 
20  plots).  The  plots  revealed  that  all  relationships  were  linear  in  nature. 

Fourth,  we  assessed  homogeneity  of  regression  by  perfonning  an  analysis  of 
variance  (ANOVA,  which  is  practically  the  same  as  an  ANCOVA  but  does  not  contain  a 
covariate  variable)  that  included  the  independent  variable  (LPTA  or  tradeoff),  each  of  the 
covariates  (NUMEVALF  ACTORS,  NUMREVIEWS,  NUMOFFERS),  and  the 
interaction  between  the  independent  variable  and  the  covariate.  The  interaction  terms 
were  not  significant,  which  indicates  that  the  relationship  between  the  DVs  (PALT  and 
CPARS  ratings)  and  the  covariates  is  the  same  at  both  levels  of  the  independent  variable 
(LPTA  or  tradeoff).  Hence,  the  assumption  of  homogeneity  of  regression  is  upheld.  The 
covariate  measuring  the  contract’s  dollar  value  (VALUE)  failed  this  test;  hence,  it  was 
dropped  from  further  analysis. 

Fifth,  we  checked  for  multicollinearity  by  assessing  the  pooled  within  cell 
tolerance  for  each  DV  (PALT  and  CPARS).  Multicollinearity  is  not  an  issue  for  the  DVs 
when  the  independent  grouping  variable  tradeoff  is  used;  however,  it  may  be  an  issue  for 
the  LPTA  group.  For  the  LPTA  group,  NUMOFFERS  is  very  highly  correlated  with 
PALT  (r  =  .94).  This  suggests  that  this  covariate  may  be  measuring  many  of  the  same 
characteristics  that  the  PALT  dependent  variable  is  measuring.  This  can  be  troublesome 
because  the  high  degree  of  similarity  makes  the  variables  redundant.  To  account  for  this 
issue,  we  chose  to  perfonn  the  MANCOVA  twice:  once  with  NUMOFFERS  included 
and  once  with  NUMOFFERS  excluded. 

Finally,  we  checked  for  homogeneity  of  covariance  matrices  between  groups  (i.e., 
sphericity)  using  the  multivariate  test  of  means  provided  in  STATA  12.1.  This  test  checks 
whether  or  not  population  variances  and  covariances  of  both  DVs  (PALT  and  CPARS) 
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are  equal  for  each  of  the  independent  variable  groups  (LPTA  and  tradeoff).  The  results 
showed  that  all  grouping  cells  were  homogenous.  With  all  assumptions  tested,  we 
perfonned  the  MANCOVA.  The  results  are  provided  in  the  following  section. 

C.  RESULTS 

With  all  the  aforementioned  assumptions  tested,  we  performed  the  MANCOVA 
analysis.  The  following  sections  outline  the  results,  first  with  NUMOFFERS  included, 
second  with  NUMOFFERS  excluded. 

1.  Results,  Including  Number  of  Offers 

The  results  show  no  instances  where  source  selection  method  (LPTA  or  tradeoff) 
produced  significantly  different  contract  outcomes.  There  were,  however,  covariates  that 
produced  different  outcomes.  Although  not  the  focus  of  our  research,  we  list  them  here 
and  urge  future  researchers  to  examine  these  variables  as  covariates,  as  they  capture  some 
of  the  variance  in  the  DV  groups.  Essentially,  both  the  number  of  evaluation  factors  a 
solicitation  contains  and  the  number  of  offers  it  receives  increase  time  to  award  (PALT). 
These  results  make  sense  because  the  PALT  includes  the  time  taken  to  select  an  awardee. 
This  selection  process  involves  evaluating  each  offer  received  in  strict  accordance  with 
the  evaluation  factors  set  forth  in  the  solicitation.  Therefore,  the  selection  process  (and 
resulting  PALT)  will  increase  as  more  offers  are  submitted  and  as  more  evaluation 
factors  are  stipulated  in  solicitations.  Table  2  shows  the  numerical  evidence  of  this  result. 
The  italicized  variables,  NUMEVALF ACTORS  and  NUMOFFERS,  were  the  only  ones 
that  demonstrated  a  statistically  significant  effect  on  PALT. 
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Table  2.  MANCOVA  Results  with  NUMOFFERS 


Variable 

Coeff 

StdErr 

t 

P>t 

95%  Cl 

PALT 

LPTATO 

-.9154339 

.513016 

-1.78 

0.100 

-2.0332 

.2023319 

#Eval 

Factors 

3.456288 

1.10602 

3.12 

0.009 *** 

1.046478 

5.866098 

#  Reviews 

.0082172 

.4027713 

0.02 

0.984 

-.8693461 

.8857806 

#  Offers 

.9032533 

.3112962 

2.90 

0.013** 

.2249971 

1.58151 

Constant 

1.507324 

1.058166 

1.42 

0.180 

-.7982225 

3.81287 

CPARS 

LPTATO 

.0447562 

.1424067 

0.31 

0.759 

-.2655213 

.3550337 

#  Eval 
Factors 

..1412446 

.3070169 

-0.46 

0.654 

-.810177 

.5276879 

#  Reviews 

.1214624 

.1118042 

1.09 

0.299 

-.122138 

.3650628 

#  Offers 

-.0865176 

.0864119 

-1.00 

0.336 

-.2747929 

.1017576 

Constant 

1.413526 

.2937335 

4.81 

0.000 

.7735361 

2.053517 

*  p<.10  **  p<.05  ***  p<.01 

Number  of  Observations  =17 

Root  Mean  Squared  Error;  PALT  =  .86  CPARS  =  .24 

R2:  PALT  =  .6266  CPARS  =  .1171 

DVs:  PALT  and  CPARS  ratings;  statistically  significant  variables  in  italics;  results 
shown  in  multivariate  regression  format 


2.  Results,  Excluding  Number  of  Offers 

Removing  the  effect  of  the  potentially  collinear  variable  NUMOFFERS,  we  find 
similar  results.  There  are  still  no  instances  where  source  selection  method  (LPT A  or 
tradeoff)  produced  significantly  different  contract  outcomes.  Number  of  Evaluation 
Factors  (NUMEVALFACTORS),  however,  still  produced  significantly  different  results 
in  PALT  (the  larger  the  number  of  factors,  the  longer  the  PALT).  This  is  shown  by  the 
italicized  row  in  Table  3. 
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Table  3 .  MANCOVA  Results  without  NUMOFFERS 


Variable 

Coeff 

StdErr 

t 

P>t 

95%  Cl 

PALT 

LPTATO 

-.1955875 

.5627742 

-0.35 

0.734 

-1.411387 

1.020212 

#Eval 

Factors 

2.660771 

1.342889 

1.98 

0.069* 

-.2403637 

5.561905 

#  Reviews 

.5984634 

.4356733 

1.37 

0.193 

-.3427515 

1.539678 

Constant 

1.921261 

1.31407 

1.46 

0.167 

-.9176158 

4.760138 

CPARS 

LPTATO 

-.0241939 

.1246599 

-0.19 

0.849 

-.2935053 

.2451175 

#  Eval 
Factors 

-.0650464 

.2974628 

-0.22 

0.830 

-.7076756 

.5775829 

#  Reviews 

.0649259 

.0965058 

0.67 

0.513 

-.1435622 

.2734141 

Constant 

1.373878 

.2910793 

4.72 

0.000 

.7450391 

2.002716 

*  p<.10  **  p<.05  ***  p<.01 

Number  of  Observations  =17 

Root  Mean  Squared  Error:  PALT  =  1.08  CPARS  =  .24 

R2:  PALT  =  .3646  CPARS  =  .0433 

DVs:  PALT  and  CPARS  ratings;  statistically  significant  variables  in  italics;  results 
shown  in  multivariate  regression  format 


D.  FURTHER  DISCUSSION 

The  previous  two  tables  show  that  evaluation  factors  and  number  of  offers 
received  are  the  only  variables  that  significantly  affect  contract  outcome.  These  two 
variables  affected  PALT,  even  though  the  CPARS  ratings  were  unaffected  (significantly) 
by  either  the  IVs  or  the  covariates.  As  a  final  analysis,  we  examined  the  relationships 
between  the  variables  themselves.  The  correlations  are  shown  in  Table  4. 
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Table  4.  Correlations 


Variable 

PALT 

CPARS 

#  Evaluation 
Factors 

#  Reviews 

#  Offers 

PALT 

1.00 

-0.3623 

0.4694 

0.5344 

0.3333 

CPARS 

0.2657 

1.00 

0.1206 

0.3161 

-0.2882 

#  Evaluation 
Factors 

0.7319 

0.2258 

1.00 

0.5799 

-0.2951 

#  Reviews 

0.3099 

0.0912 

-0.1722 

1.00 

0.3650 

#  Offers 

0.9370 

0.4951 

0.6139 

0.4753 

1.00 

Non-italicized,  lower  triangle=LPTA;  italicized,  upper  triangle=tradeoff 


These  correlations  confirm  the  previous  findings,  and  should  come  as  no  surprise 
to  anyone  who  has  taken  part  in  a  source  selection.  For  example,  the  highest  overall 
correlation  exists  between  PALT  and  NUMOFFERS  (r=.9370),  and  between  PALT  and 
NUMEVALFACTORS  (r=.7319)  in  an  LPTA  setting.  This  suggests  that  the  bulk  of  the 
PALT  is  taken  up  by  reviewing  offers;  the  more  offers  that  are  submitted,  the  more  time 
it  will  take  to  review  those  offers  (thus  increasing  PALT).  There  is  a  high  correlation 
between  PALT  and  evaluation  factors  (second  highest  correlation  found  through 
correlation  analysis).  These  two  aforementioned  high  correlations  demonstrate  that  there 
is  potentially  a  large  compounding  effect  of  evaluation  factors  and  number  of  offers  on 
PALT. 

The  highest  correlation  found  in  the  tradeoff  setting  exists  between  evaluation 
factors  and  NUMREVIEWS  (r=.5799).  The  same  amount  of  strong  correlation  between 
these  two  covariates  was  not  noticed  in  the  LPTA  situation.  These  results  also  make 
sense;  for  tradeoff  scenarios,  source  selection  is  more  complex  as  each  evaluation  factor 
is  weighed  as  a  possible  basis  for  award.  The  more  factors  present  as  possible  bases  for 
award,  the  more  complex  this  determination  becomes;  thus,  the  process  will  likely  require 
more  reviews.  In  contrast,  with  LPTA,  the  decision  ultimately  comes  down  to  price 
(provided  the  lowest  priced  offer  is  technically  acceptable),  and  the  process  is  more 
straightforward,  thus  requiring  fewer  reviews.  Other  noteworthy  correlations  in  the 
tradeoff  setting  include  the  correlation  between  PALT  and  NUMEVALFACTORS 
(r=.4694)  and  also  between  PALT  and  NUMREVIEWS  (r=.5344).  These  correlations 
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confirm  that  an  increase  in  evaluation  factors,  and/or  an  increase  in  the  number  of 
reviews,  will  increase  the  PALT.  The  fact  that  this  can  be  shown  statistically  should 
affirm  the  need  to  eliminate  unnecessary  levels  of  review  or  evaluation  factors.  Any 
increase  to  PALT  should  be  for  worthwhile  evaluation. 

It  is  also  worth  noting  the  highest  instance  of  negative  correlation  found  in  the 
analysis.  This  exists  between  the  two  DVs,  PALT  and  CPARS  (r=-0.3623).  The  possible 
reasons  for  this  observation  are  numerous,  but  the  following  scenarios  provide  possible 
explanations.  A  low  PALT  could  mean  that  the  contract  awardee  was  quickly  selected  as 
a  clear  winner  and  that  this  contractor  lived  up  to  expectations  by  perfonning  well. 
Conversely,  a  high  PALT  suggests  that  the  decision  was  more  ambiguous,  with  no  clear 
winner.  A  decision  made  under  the  latter  conditions  would  be  more  likely  to  result  in 
selecting  a  less-than-optimal  contractor.  While  this  is  educated  speculation,  more 
research  needs  to  be  conducted  to  confirm  or  negate  this  notion. 

One  last  important  observation  from  the  correlation  table  is  the  positive 
correlation  between  number  of  offers  and  CPARS  ratings  in  LPTA  source  selections 
(r=.4951).  This  correlation  helps  make  the  case  that  competition  produces  better 
outcomes.  While  nothing  can  be  stated  for  certain  here,  this  suggests  that  higher  levels  of 
competition  will  lead  to  the  selection  of  high-performing  contractors. 

E.  RECOMMENDATIONS  BASED  ON  FINDINGS 

We  are  unable  to  make  concrete  recommendations  for  change  based  on  our 
limited  amount  of  data.  The  one  recommendation  we  can  make  is  that  each  requirement 
be  fully  understood  in  order  to  make  the  best  decision  as  to  whether  the  LPTA  or  tradeoff 
source  selection  method  is  more  appropriate.  In  the  solicitation  planning  phase, 
contracting  personnel  together  with  their  customers  should  closely  examine  the  number 
of  evaluation  factors  and  the  number  of  planned  reviews,  with  the  understanding  that 
these  variables  will  affect  PALT.  This  recommendation  mirrors  that  of  contract 
management  expert  Marge  Rumbaugh  to  “limit  evaluation  factors  to  those  that  are  critical 
to  program  success”  (Rumbaugh,  2015,  p.  12).  The  likely  additional  PALT  should  be 
weighed  against  the  need  for  certain  evaluation  factors  and  reviews  to  determine  which  is 
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more  important  to  the  acquisition.  In  other  words,  the  acquisition  risk  posed  by 
conducting  fewer  reviews  should  be  weighed  against  that  of  increased  PALT.  These 
recommendations  also  echo  the  policy  memorandum  issued  by  Shay  Assad,  then-director 
of  Defense  Procurement  and  Acquisition  Policy  (2011). 

F.  SUMMARY 

These  results  have  served  to  mathematically  confirm  some  basic  assumptions  of 
the  source  selection  process.  Also,  the  idea  that  one  source  selection  method  may  lead  to 
better  outcomes  than  another  was  unsubstantiated  by  our  analyses.  We  encourage  further 
research  efforts  in  this  area  to  expand  on  these  findings  and  possibly  find  other  ways  of 
predicting  or  improving  contract  outcomes.  Having  shown  the  results  of  our  data 
analysis,  in  the  next  chapter,  we  draw  conclusions  and  present  recommendations,  as  well 
as  answer  our  research  questions. 
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V.  SUMMARY,  CONCLUSION,  AND  AREAS  FOR  FURTHER 


RESEARCH 


A.  INTRODUCTION 

The  previous  chapter  included  a  brief  discussion  of  some  of  the  observed  effects 
of  source  selection  method  on  contract  outcomes,  namely  PALT  and  CPARS  ratings.  In 
this  chapter,  we  summarize  our  findings  and  present  our  conclusion  from  the  data 
analysis  found  in  Chapter  IV,  answer  the  original  research  questions,  and  make 
suggestions  for  future  research. 

B.  SUMMARY 

The  goal  of  this  research  was  to  determine  whether  or  not  there  is  a  relationship 
between  the  source  selection  methods,  and  the  contract  outcomes  of  PALT  and  CPARS 
ratings.  We  reviewed  many  professional  journal  articles  and  Government  Accountability 
Office  reports  to  better  understand  both  methods.  We  also  perfonned  hands-on  research 
by  reviewing  33  contract  files  at  Peterson  AFB  and  Schriever  AFB  and  collecting  the 
data  points  indicated  in  Chapters  III  and  IV;  then  we  utilized  statistical  analysis  to 
determine  the  effects  of  those  data  points  on  PALT  and  CPARS  ratings.  Our  data 
collection  method  produced  a  sample  size  that  was  adequate  for  our  purposes  but  too 
small  to  influence  policy  decisions  regarding  source  selection.  We  have  made  some 
suggestions  regarding  information  provided  by  our  analysis,  with  the  recommendation 
that  our  study  results  be  expanded  upon  and  retested  with  future  research. 

The  main  takeaway  from  our  research  is  that  PALT  and  CPARS  are  more  affected 
by  number  of  evaluation  factors  and  number  of  offers  received  than  they  are  by  the 
source  selection  method  used.  The  circumstances  of  each  acquisition  are  different,  and 
COs  must  exercise  expert  judgment  in  determining  which  source  selection  method  to  use. 
Both  methods  will  prove  successful  as  long  as  they  are  executed  properly.  Proper 
execution  will  work  towards  meeting  the  agency’s  contracting  needs  and  increasing 
mission  success  while  also  safeguarding  limited  government  funds. 
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c. 


CONCLUSION 


The  limitation  of  our  research  was  that  the  sample  size  we  collected  was  small; 
thus,  our  conclusions  must  be  read  with  that  caveat  in  mind.  The  data  analysis  showed 
that  the  source  selection  process  used  did  not  significantly  affect  PALT  or  CPARS 
ratings.  Only  the  number  of  evaluation  factors  and  the  number  of  offers  received 
produced  a  statistically  significantly  effect.  Correlation  analysis  confirmed  the  finding 
that  the  number  of  evaluation  factors  and  the  number  of  offers  were  positively  correlated 
to  PALT,  indicating  that  PALT  increases  along  with  an  increase  in  either  factor. 

D.  ANSWERING  THE  RESEARCH  QUESTIONS 

In  Chapter  I  of  this  research,  we  posed  the  following  questions: 

1 .  What  is  the  relationship  between  the  source  selection  method  (LPTA 
versus  tradeoff)  and  CPARS  ratings? 

2.  What  is  the  relationship  between  the  source  selection  method  (LPTA 
versus  tradeoff)  and  PALT? 

3.  What  percentage  of  procurements  using  the  tradeoff  source  selection 
method  is  awarded  based  on  non-price  factors? 

Questions  1  and  2  can  be  answered  concurrently.  The  underlying  finding  of  the 
data  analysis  was  that  the  source  selection  process  used  did  not  significantly  affect  PALT 
or  CPARS  ratings.  Only  the  number  of  evaluation  factors  and  the  number  of  offers 
received  significantly  affected  PALT.  In  other  words,  PALT  will  increase  as  complexity 
of  the  acquisition  increases.  This  is  not  to  say,  of  course,  that  evaluation  factors  should  be 
minimized  in  order  to  limit  PALT.  If  an  acquisition  is  worth  conducting,  it  is  worth 
taking  the  time  to  do  it  correctly;  there  is  no  sense  in  sacrificing  quality  of  source 
selection  to  reduce  lead  time. 

Contract  performance,  in  the  form  of  CPARS  ratings,  was  not  affected  by  any  of 
the  variables  tested.  This  finding  suggests  that  both  source  selection  methods  result  in 
equal  performance.  From  a  common  sense  standpoint,  this  result  makes  sense,  as  there  is 
no  reason  to  suspect  that  contractors  chosen  with  tradeoffs  are  inherently  better  than 
those  chosen  with  the  LPTA  method — neither  would  the  quality  of  contractor 
performance  be  expected  to  change  as  a  result  of  variables  such  as  the  number  of 
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evaluation  factors.  A  contractor’s  perfonnance  is  not  tied  to  the  procedures  the  agency 
uses  to  select  them  for  contract  award. 

A  reminder  of  our  small  sample  size  must  accompany  the  answer  to  Question  3. 
Of  the  23  contracts  we  sampled  that  used  the  tradeoff  method,  eight  were  awarded  based 
on  non-price  factors.  In  other  words,  factors  other  than  price  were  the  basis  for  award  for 
34.8%  of  our  tradeoff  sample  contracts.  The  remaining  65.2%  ended  up  being  awarded 
based  on  price,  even  though  the  tradeoff  method  was  used.  This  finding  is  in  line  with  the 
Government  Accountability  Office’s  finding,  presented  in  our  literature  review;  namely, 
that  the  government  selected  higher  priced  offers  in  less  than  a  quarter  of  tradeoff 
examples  observed  (GAO,  2010).  This  is  an  interesting  finding,  although  a  much  larger 
and  varied  sample  size  (i.e.,  from  several  installations)  would  be  needed  in  order  to  draw 
any  real  conclusions  about  common  practices. 

E.  AREAS  FOR  FURTHER  RESEARCH 

A  consistent  challenge  throughout  our  project  was  ensuring  we  would  have 
enough  data  points  to  ensure  adequate  power  for  our  statistical  analysis.  Our  limited 
sample  size  opens  the  door  for  future  researchers  to  collect  more  data,  in  the  same 
manner  as  our  study,  to  re-run  the  model.  Specifically,  we  recommend  that  further 
research  of  this  nature  be  conducted  at  other  operational  bases  within  the  Air  Force.  From 
there,  the  research  should  expand  to  major  weapon  system  acquisition  bases.  After  that 
research  has  been  conducted,  the  other  military  branches  should  duplicate  these  research 
efforts,  both  at  the  operational-  and  systems-levels.  As  stated  previously,  our  hope  is  that 
this  research  serves  as  a  starting  point  for  a  larger  effort  that  eventually  encompasses  the 
entire  defense  acquisition  field.  With  such  a  large  data  sample,  the  model  will  obtain 
adequate  statistical  power.  Such  a  development  would  allow  future  researchers  to  draw 
more  accurate  and  definitive  conclusions  as  to  the  relationship  between  source  selection 
method  and  contract  outcomes.  At  that  point,  the  findings  may  point  to  areas  where  the 
source  selection  process  can  be  improved  and  cost  or  process  savings  realized  across  the 
Department  of  Defense. 
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